Need someone help to add CAsT 2020 baseline results:

• [ ] Naive: CQR without canonical responses
• [ ] Canonical: CQR with canonical (manual) response

CQR methods: HQE /Ntr (T5)

Dear authors,

I am trying to use your method in some of my work. For that, I need to get the reformulated queries (instead of only the generated ranked hits).

I am trying to run the HQE experiment as indicated using:

python -m experiments.run_retrieval \
      --experiment hqe \
      --hits 1000 \
      --sparse_index cast2019 \
      --qid_queries $input_query_json \
      --output ./output/hqe_bm25 

However, when I print the arguments passed inside the retrieval pipeline (L101 of retrieval_pipeline.py) I get as query the raw/original/last-turn query string, and as manual_context_buffer[turn_id] simply None. If I'm not mistaken, that means that running the specific experiment equals to no reformulation being done at all. Can you check/confirm this?

Digging more into the code, it seems to me that the queries I'd like to access are inside cqr_queries, but still, it seems to me that context should be empty/None in that case - probably resulting to no reformulation done at all.

Thank you with the project.

The fix to below will be hits = rp.retrieve(query, manual_context_buffer[turn_id-1] if turn_id!=0 else None), to pass the last previous canonical response.

https://github.com/castorini/chatty-goose/blob/f9c21c8b7b6194d11d7aec5b4e218174cde98418/experiments/run_retrieval.py#L100

Main change:

1. change --dense_index from temporary one to pyserini prebuilt index name
2. fix canonical response bug, which previously add current response to context
3. since now we have --dense_index, change option name --index to --sparse_index

New features added: (only for T5 and CQE, may consider HQE in the future) (1) Dense retrieval (2) Dense-sparse hybrid retrieval

Some arg might be confused and may be changed in the future: (1) --index, --dense_index: may change to --sparse_index and --dense_index (2) --experiment now has options (hqe,cqe,t5,fusion,cqe_t5_fusion) may change to (hqe,cqe,t5,hqe_t5fusion,cqe_t5_fusion)

This PR adds both naive and canonical baselines for CAsT2020 topics. The results are overall lower as compared to CAst2019 and the results from the canonical run are only slightly better for some metrics as compared to results from the naive run.

Resolves #23

This PR adds support for using manual_canonical_result_id in the CAsT2020 data for both ntr and hqe (for #23).

For ntr, rewrite uses the passage corresponding to the canonical document in the history. We only use 1 passage in the historical context as otherwise, it exceed 512 tokens limit. For e.g., it uses q1/P1/q2 and then q1/q2/P2/q3 and so on.

Add CQR replication for Fusion BM25

Library versions used: torch==1.7.0 torchvision==0.8.1 torchtext==0.8

Results:

map                   	all	0.2584
recall_1000           	all	0.8028
ndcg_cut_1            	all	0.3353
ndcg_cut_3            	all	0.3247

Details and reproduction results can be found in the notebook

Breaking changes:

Renamed several classes to follow Python conventions / be more consistent

• chatty_goose.agents.cqragent -> chatty_goose.agents.chat
• HQE -> Hqe
• T5_NTR -> Ntr
• HQESettings -> HqeSettings
• T5Settings -> NtrSettings
• CQRType -> CqrType
• CQRSettings -> CqrSettings
• CQR -> ConversationalQueryRewriter

With a fresh install, we get the following error if we try to run anything:

OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.

Solution is:

$ python -m spacy download en_core_web_sm

We should document this.

With a from-scratch installation, the module pulls in Torch 1.8, which causes this error:

ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler' (/anaconda3/envs/chatty-goose-test/lib/python3.7/site-packages/torch/optim/lr_scheduler.py)

Downgrading fixes the issue:

$ pip install torch==1.7.1 torchtext==0.8.1

Should we pin the version in our module dependencies? Or at the very least this needs to be documented.=

Hi,

When I install chatty-goose from github using:

python -m pip install git+https://github.com/castorini/chatty-goose.git

I met this issue:

ERROR: Cannot install chatty-goose and chatty-goose==0.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    chatty-goose 0.2.0 depends on pyserini==0.14.0
    pygaggle 0.0.3.1 depends on pyserini==0.10.1.0

It seems that chatty-goose requires pyserini==0.14.0 as well as pygaggle 0.0.3.1. However, pygaggle 0.0.3.1 and pyserini==0.14.0 do not play nice with each other

Could someone provide some help?

Thanks!

Does it make sense to expand Chatty Goose to new datasets? For example:

• MANtIS - a multi-domain information seeking dialogues dataset: https://guzpenha.github.io/MANtIS/
• ClariQ - Search-oriented Conversational AI (SCAI) EMNLP https://github.com/aliannejadi/ClariQ

According to @edwinzhng's replication log, we have a reranker checkpoint mismatch issue. Currently, we have diffs in our reranking model and the pygaggle's default model.

Related to this issue: I think we need a folder to put/track our tf2torch ckpt transformation/sanity check scripts?