Add Marengo embedding predictor for zero-shot moment retrieval

[mohit-twelvelabs]

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

This PR adds an optional, zero-shot moment retrieval predictor backed by TwelveLabs Marengo embeddings.

What it adds

• lighthouse/marengo_predictor.py — a MarengoPredictor that segments and embeds a video server-side, embeds the text query into the same 512-dim Marengo space, and ranks clips by cosine similarity. Its output matches the existing predictors: {"pred_relevant_windows": [[start, end, score], ...]}.
• Re-exported from lighthouse.models alongside the existing predictors.
• api_example/marengo_demo.py mirroring the existing demo.py / amr_demo.py.
• A README section, a marengo optional extra in setup.py, a mypy.ini entry, and tests.

Why it helps Lighthouse

• A training-free baseline that needs no local checkpoint, feature files, or GPU.
• Not bound by the 150s benchmark limit, since embedding happens server-side — handy for longer videos.

Opt-in and non-breaking

• The twelvelabs SDK is an optional extra (pip install 'lighthouse[marengo]') and is only imported when MarengoPredictor is instantiated, so existing users pay no import or dependency cost. No defaults or existing predictors change.

How it was tested

• No-network unit tests in tests/test_marengo.py cover the cosine-similarity ranking and the empty-input path.
• A live test (skipped unless TWELVELABS_API_KEY is set) asserts a Marengo text embedding is 512-dim — verified passing locally against the API.
• ruff and mypy pass on the new files.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Per the contributing note, happy to open a tracking issue first if you'd prefer to discuss before reviewing — opening the PR so the proposed change is concrete.

[awkrail]

Thank you for your contribution. We will look into the code today.

[awkrail]

Does this work well? As shown in README, lighthouse is installed from github repo directly. So, correctly, pip install 'lighthouse[marengo] @ git+https://github.com/line/lighthouse.git' works?

[awkrail]

In addition, we would like to think about the naming. Now, Twelvelabs have Marengo as a video embedding model, but in the future, Twelvelabs may release other video embedding models, right? If so, I would like to use lighthouse[twelvelabs] rather than Marengo itself.

[awkrail]

@mohit-twelvelabs I added some comments to your code. I will check the logit part tomorrow by running the code...

[mohit-twelvelabs]

Thanks for the careful review, @awkrail! Both great points — addressed in f3ee766:

1. Install command — you're right, since lighthouse installs from GitHub (not PyPI), pip install 'lighthouse[marengo]' wouldn't resolve. Fixed the README to the git form you suggested:

   pip install 'lighthouse[twelvelabs] @ git+https://github.com/line/lighthouse.git'
   

2. Naming — fully agree on future-proofing. I renamed the optional extra from [marengo] to [twelvelabs] (it installs the twelvelabs SDK, which will cover future embedding models too), so the install is now lighthouse[twelvelabs].

I left the predictor class as MarengoPredictor since it's specific to the Marengo embedding model, but happy to rename it (e.g. TwelveLabsPredictor with a model_name arg) if you'd prefer the vendor-scoped naming there as well — just say the word.

No rush on the logit part — thanks for offering to run it, and let me know if anything comes up when you do.

[awkrail]

@mohit-twelvelabs
Hi, thank you for resolving the install command.
In terms of naming, I like TwelveLabsPredictor for the future update and give model_name to the class instance. So, could you change MarengoPredictor to TwelveLabsPredictor?

[awkrail]

I will check the logic part today.

[mohit-twelvelabs]

Done — renamed MarengoPredictor -> TwelveLabsPredictor and the embedding model is now set via a model_name instance param (defaults to marengo3.0), so it's ready to support future TwelveLabs embedding models without code changes. I updated the README usage example and every reference (import in lighthouse/models.py, the api_example demo, and the tests). This is in commit 54b5adc.

Thanks for the suggestion — I like this framing much better for the future update. I'll keep an eye out for your review of the logic part.

[awkrail]

@mohit-twelvelabs
Thank you for fixing. Could you rename filename from marengo_predictor.py to twelvelabs_predictor.py?

In addition, I cloned the head of this PR and got two errors and one question:
First error: model: TwelveLabsPredictor = TwelveLabsPredictor(api_key=api_key, clip_length=2.0)

Traceback (most recent call last):
  File "/dlrs/home/codes/lighthouse/api_example/marengo_demo.py", line 28, in <module>
    video: Dict[str, List[List[float]]] = model.encode_video('api_example/RoripwjYFp8_60.0_210.0.mp4')
  File "/opt/conda/lib/python3.10/site-packages/lighthouse/marengo_predictor.py", line 104, in encode_video
    task = self._client.embed.tasks.create(
  File "/opt/conda/lib/python3.10/site-packages/twelvelabs/embed/tasks/client.py", line 226, in create
    _response = self._raw_client.create(
  File "/opt/conda/lib/python3.10/site-packages/twelvelabs/embed/tasks/raw_client.py", line 262, in create
    raise BadRequestError(
twelvelabs.errors.bad_request_error.BadRequestError: headers: {'date': 'Thu, 09 Jul 2026 01:28:07 GMT', 'content-type': 'application/json',
'content-length': '125', 'connection': 'keep-alive', 'content-encoding': 'gzip', 'tl-report': 'backend', 'vary': 'Accept-Encoding', 'x-ratel
imit-dimensions': 'Duration,Request', 'x-ratelimit-duration-limit': '36000', 'x-ratelimit-duration-remaining': '35999', 'x-ratelimit-duratio
n-reset': '1783564087', 'x-ratelimit-request-limit': '25', 'x-ratelimit-request-remaining': '24', 'x-ratelimit-request-reset': '1783560547',
 'x-trace-id': '1e7852cecf276a6d3cf71e08f45d5c3d'}, status_code: 400, body: {'message': {'code': 'parameter_invalid', 'message': 'The video_
clip_length parameter is invalid. received string, but expected integer'}}

When changing 2.0 to 2, this error doesn't appear, so this embedding construction supports only integer length, right? If so, could you fix from 2.0 to 2?

After resolving this issue, I met the second issue when running .encode() function

Traceback (most recent call last):
  File "/dlrs/home/codes/lighthouse/api_example/marengo_demo.py", line 28, in <module>
    video: Dict[str, List[List[float]]] = model.encode_video('api_example/RoripwjYFp8_60.0_210.0.mp4')
  File "/opt/conda/lib/python3.10/site-packages/lighthouse/marengo_predictor.py", line 104, in encode_video
    task = self._client.embed.tasks.create(
  File "/opt/conda/lib/python3.10/site-packages/twelvelabs/embed/tasks/client.py", line 226, in create
    _response = self._raw_client.create(
  File "/opt/conda/lib/python3.10/site-packages/twelvelabs/embed/tasks/raw_client.py", line 262, in create
    raise BadRequestError(
twelvelabs.errors.bad_request_error.BadRequestError: headers: {'date': 'Thu, 09 Jul 2026 01:28:51 GMT', 'content-type': 'application/json',
'content-length': '169', 'connection': 'keep-alive', 'content-encoding': 'gzip', 'tl-report': 'backend', 'vary': 'Accept-Encoding', 'x-ratel
imit-dimensions': 'Duration,Request', 'x-ratelimit-duration-limit': '36000', 'x-ratelimit-duration-remaining': '35999', 'x-ratelimit-duratio
n-reset': '1783564087', 'x-ratelimit-request-limit': '25', 'x-ratelimit-request-remaining': '24', 'x-ratelimit-request-reset': '1783560547',
 'x-trace-id': 'aedc69859077963a4f4af38cfeb05ffa'}, status_code: 400, body: {'code': 'video_resolution_too_low', 'message': 'The resolution
of the video is too low. Please upload a video with resolution between 360p(360x360) and 2160p(5184x2160). Current resolution is 534x300.'}

Using RoripwjYFp8_60.0_210.0.mp4 doesn't work, so I upscale it via ffmpeg -i api_example/RoripwjYFp8_60.0_210.0.mp4   -vf "scale='if(gt(a,1),-2,360)':'if(gt(a,1),360,-2)'"  -c:a copy api_example/RoripwjYFp8_60.0_210.0_360p.mp4 and this error resolves. Could you upscale the example video and put it in api_example/? This is helpful for users.

Finally, the result is:

{'pred_relevant_windows': [[6.0, 12.0, 0.4402], [42.0, 48.0, 0.4402], [48.0, 54.0, 0.4402], [96.0, 102.0, 0.108.0, 114.0, 0.4402], [144.0, 150.0165, 0.3655], [12.0, 18.0, 0.3446], [72.0, 78.0, 0.3283], [60.0, 66.0, 0]}

When checking the video with this result, the performance seems not to be good unfortunately. I understood this task is zero-shot so difficult compared with other trained models, so performance matter is OK. However, I am unsure whether this result is expected or not, so could you double-check the result?

[awkrail]

About the result, I found that the clip_length is 2 but the output seems to be 6 sec, right? I am unsure why this difference occurs inside the sever side.