When I tried to run the example case like this

python hf_bart_export.py
python ls_bart.py

It has some errors

initializing bart tokenizer...
creating lightseq model...
Traceback (most recent call last):
  File "ls_bart.py", line 102, in <module>
    main()
  File "ls_bart.py", line 69, in main
    ls_model = lsi.Transformer("lightseq_bart_base.hdf5", 128)
RuntimeError: Parse weights from [lightseq_bart_base.hdf5] failed.

Alright，I tried to run other case , huggingface gpt2 in examples:

python hf_gpt2_export.py
python ls_gpt.py

It had some error again:

initializing gpt tokenizer...
Downloading: 100%|███████████████████████████████████████████████████████| 1.04M/1.04M [00:00<00:00, 1.81MB/s]
Downloading: 100%|█████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 1.36MB/s]
Downloading: 100%|███████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 2.29MB/s]
lightseq tokenizer pad token id: 50257
huggingface tokenizer pad token id: 50256
creating lightseq model...
Traceback (most recent call last):
  File "ls_gpt.py", line 119, in <module>
    main()
  File "ls_gpt.py", line 79, in main
    ls_model = lsi.Gpt("lightseq_gpt2_base.hdf5", max_batch_size=16)
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. lightseq.inference.Gpt(weight_path: str, max_batch_size: int, max_step: int)

Invoked with: 'lightseq_gpt2_base.hdf5'; kwargs: max_batch_size=16

I don't know how to fixed them. Can you give me some advices. thank you very much.

Hi, I used the python export/ls_fs_transformer_export.py to export lightseq trained NMT model to do inference, but I found the result is quiet abnormal. These are some details output in the ls_fs_transformer_export.py test part.

generator config beam size: 4 extra decode length(max decode length - src input length): 50 length penalty: 0.6 diverse lambda: 0 sampling method: beam_search topk: 1 topp: 0.75 Allocated 882MB GPU buffer for transformer decoder buffer init start decoder buffer init succeed pb results: (array([[[ 4, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 6]]], dtype=int32), array([[0.]], dtype=float32)) hdf5 results: (array([[[ 4, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 164, 6]]], dtype=int32), array([[0.]], dtype=float32))

I also tested more examples, and it continued to generate some repeated logits, and when I decoded the array with my tgt_dict, it generated something like this:

thesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesamesame.

I used the 0.10.2 fairseq and 2.1.4 lightseq version, and the lightseq-generate result seems normal. I think maybe something wrong happened in the export procedure. Looking forward for your reply.

Fixed the implementation of position embedding

• the size of position matrix is determined by max_positions parameter
• ignore all the padding tokens when calculating the token position
• the position index begin from padding_idx + 1, consistent with fairseq implementation

I use the code below to test the vit block speed. The output shows the speed is almost the same between pytorch and lightseq

Did I missed something?

Output for forward only:

timm finished 500 running, avg_time: 76.379987 ms light_seq finished 500 running, avg_time: 75.543549 ms

The output for forward + backward:

timm finished 500 running, avg_time: 228.803998 ms light_seq finished 500 running, avg_time: 227.007331 ms

from timm.models.vision_transformer import Block
from lightseq.training.ops.pytorch.transformer_encoder_layer import LSTransformerEncoderLayer
from easydict import EasyDict as edict
import torch.nn as nn
import torch
import time
import sys
sys.path.append('./')


torch.backends.cudnn.benchmark = True


def generate_dummy_data(args):
    inputs = torch.randn([args.bs, args.num_token, args.dim]).cuda()
    return (inputs, )


def get_timm_block(args):
    return Block(
        dim=args.dim,
        num_heads=args.num_heads,
        mlp_ratio=args.mlp_ratio,
        qkv_bias=False,
        drop=False,
        attn_drop=False,
        init_values=None,
        drop_path=0,
        act_layer=nn.GELU,
        norm_layer=nn.LayerNorm
    )

class LSBlockWrapper(LSTransformerEncoderLayer):
    def forward(self, x):
        B, N, C = x.shape
        mask = torch.zeros([B, N, N], device=x.device, dtype=x.dtype)
        return super().forward(x, mask)

def get_ls_block(args):
    config = LSBlockWrapper.get_config(
        max_batch_tokens=args.num_token * args.bs,
        max_seq_len=args.num_token,
        hidden_size=args.dim,
        intermediate_size=int(args.mlp_ratio * args.dim),
        nhead=args.num_heads,
        attn_prob_dropout_ratio=0,
        hidden_dropout_ratio=0,
        activation_dropout_ratio=0,
        pre_layer_norm=True,
        fp16=False,
        local_rank=0,
        activation_fn='gelu')
    return LSBlockWrapper(
            config=config,
            initial_weights=None,
            initial_biases=None
        )


def run(module, args, name='Unknown'):
    inputs = generate_dummy_data(args)

    # cudnn warmup
    for _ in range(50):
        if args.backward:
            module(*inputs).sum().backward()
        else:
            module(*inputs)

    torch.cuda.synchronize()
    t0 = time.time()

    for _ in range(args.num_iter):
        if args.backward:
            module(*inputs).sum().backward()
        else:
            module(*inputs)

    torch.cuda.synchronize()
    t1 = time.time()

    avg_time = (t1 - t0) * 1000 / args.num_iter
    print(
        f'>>> {name} finished {args.num_iter} running, avg_time: {avg_time:.6f} ms')
    return avg_time


def main():
    args = edict()
    args.num_iter = 500
    args.backward = False

    args.bs = 128
    args.dim = 1280
    args.num_heads = 16
    args.mlp_ratio = 4.0
    args.num_token = 256

    timm_block = get_timm_block(args).cuda()
    ls_block = get_ls_block(args).cuda()

    run(timm_block, args, name='timm')
    run(ls_block, args, name='light_seq')

    print('Finished.')

if __name__ == '__main__':
    main()

when I use lightseq(2.0) export gpt2-large, it raises an error ValueError: Message Gpt exceeds maximum protobuf size of 2GB: 3096122166

hf_gpt2_export.py is as follows

if __name__ == "__main__":
    output_lightseq_model_name = "lightseq_gpt2_large.pb"
    input_huggingface_gpt_model = "gpt2-large"
    head_number = 36
    # generation_method should be "topk" or "topp"
    generation_method = "topk"
    topk = 1
    topp = 0.75
    # default eos_id from https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel
    eos_id = 50256
    pad_id = 50257
    extract_gpt_weights(
        output_lightseq_model_name,
        input_huggingface_gpt_model,
        head_num=head_number,  # layer number
        generation_method=generation_method,
        topk=topk,
        topp=topp,
        eos_id=eos_id,
        pad_id=pad_id,
    )

['transformer.h.34.mlp.c_proj.bias'] -> ffn_second_bias, shape: (1280,), convert finished.
['transformer.h.35.ln_1.weight'] -> multihead_norm_scale, shape: (1280,), convert finished.
['transformer.h.35.ln_1.bias'] -> multihead_norm_bias, shape: (1280,), convert finished.
['transformer.h.35.attn.c_attn.weight'] -> multihead_project_kernel_qkv, shape: (1280, 3840), convert finished.
['transformer.h.35.attn.c_attn.bias'] -> multihead_project_bias_qkv, shape: (3840,), convert finished.
['transformer.h.35.attn.c_proj.weight'] -> multihead_project_kernel_output, shape: (1280, 1280), convert finished.
['transformer.h.35.attn.c_proj.bias'] -> multihead_project_bias_output, shape: (1280,), convert finished.
['transformer.h.35.ln_2.weight'] -> ffn_norm_scale, shape: (1280,), convert finished.
['transformer.h.35.ln_2.bias'] -> ffn_norm_bias, shape: (1280,), convert finished.
['transformer.h.35.mlp.c_fc.weight'] -> ffn_first_kernel, shape: (1280, 5120), convert finished.
['transformer.h.35.mlp.c_fc.bias'] -> ffn_first_bias, shape: (5120,), convert finished.
['transformer.h.35.mlp.c_proj.weight'] -> ffn_second_kernel, shape: (5120, 1280), convert finished.
['transformer.h.35.mlp.c_proj.bias'] -> ffn_second_bias, shape: (1280,), convert finished.
['transformer.ln_f.weight'] -> norm_scale, shape: (1280,), convert finished.
['transformer.ln_f.bias'] -> norm_bias, shape: (1280,), convert finished.
['transformer.wte.weight'] -> token_embedding, shape: (50257, 1280), convert finished.
['transformer.wpe.weight'] -> position_embedding, shape: (1024, 1280), convert finished.
Wrting to lightseq_gpt2_large.pb
Traceback (most recent call last):
  File "hf_gpt2_export.py", line 127, in <module>
    pad_id=pad_id,
  File "hf_gpt2_export.py", line 100, in extract_gpt_weights
    fout.write(gpt.SerializeToString())
ValueError: Message Gpt exceeds maximum protobuf size of 2GB: 3096122166

hi, I have 1 question. When a large amount of text requests the model, the model starts to run properly. After the model runs for a period of time, the program reports an error : [CUDA][ERROR] /tmp/build-via-sdist-uagdfpbf/lightseq-2.2.1/lightseq/inference/pywrapper/gpt.cc.cu(160): misaligned address.

Hi guys,

Two questions related beam search confused me and I am looking forward to reply😊。

1. Your beam search is same as T2T?
2. length_penalty == 1.0 means no length_penalty?

Thx

I tried to set up LightSeq on docker system (RTX2080TI 4-way or A100 2-way) but failed to set it for 8 hours.

Therefore, please upload Dockerfile or Images using LightSeq system test.

(I tested based on image file nvcr.io/nvidia/pytorch:21.08, 20.12, 20.10, taka23/lightseq .. etc but didn't succeed.)

Hello, thank you for your contribution. I want to replace the encoders in vit-small with LSHFTransformerEncoderLayer. For each encoder, num_attention_heads = 6 and hidden_dim = 384. However, there is an error here says that hidden_dim must be an integer multiple of 256. Why does LSHFTransformerEncoderLayer have this restriction？ Is there any solution to use LSHFTransformerEncoderLayer in vit-small? Correct me if I am wrong. Thanks!

Hi, i could not find any details about lightseq model inference for fairseq task after training, did i miss something? I mean after training, the model arch is ls_tranformer, i can't use native fairseq-generate command for inference, and i don't find something like lightseq-generate. I find the examples about inference are huggingface models such as bart and gpt2, and no after-training fairseq model inference documents are provided. Could someone tell me how to do this?

I am curious of trying LightSeq to speed up my inference for a vanilla Transformer Encoder-Decoder (Vasawani 17) model. My original model was trained with FairSeq (or OpenNMT-py). Is there any example or places that you can refer to help me convert my transformer model to the format compatible of running LightSeq?

I try to compile lightseq by using build.sh, but run into the following problem:

lightseq/csrc/proto/bert_weight.cc:451:15: error: ‘class Bert’ has no member named ‘ParseFromIstream’; did you mean ‘ParseFromString’?
     if (!bert.ParseFromIstream(&raw_input)) {
               ^~~~~~~~~~~~~~~~
               ParseFromString

lightseq/csrc/proto/bert_crf_weight.cc:38:37: error: no match for ‘operator[]’ (operand types are ‘const google::protobuf::RepeatedPtrField<BertCrfEncoderLayer>’ and ‘int’)
   _inner_size = bert.encoder_stack()[0].ffn_first_kernel_size() / _hidden_size;

Branch master and tag v3.0.1 both failed.

Did I miss something? How can I manage to compile this project?

The constructor creates new objects without shared_ptrs, but the destructor is empty.

In cpp:

DecSelfAttentionLayer<T1, T2>::DecSelfAttentionLayer(
    int layer_id, int max_batch_tokens, int max_seq_len, int hidden_size,
    int num_heads, float attn_prob_dropout_ratio,
    float hidden_output_dropout_ratio, bool pre_or_postLayerNorm,
    bool is_post_ln, bool is_continuous_cache)
    : Layer("DecSelfAttentionLayer"),  // necessary
      _layer_id(layer_id),
      _max_batch_tokens(max_batch_tokens),

     ..............................
      // operators
      _attn_ln(
          new LayerNormalizeOp<T1, T2>(max_batch_tokens, hidden_size, false)),

In header: virtual ~DecSelfAttentionLayer() {}

Not sure if this is by design or missing delete calls in the destructor.

In lightseq/csrc/models/transformer.cu, Should cache_k_out and cache_v_out call set_ancestor? Otherwise why not remove the unused variable cache_k_out and cache_k_out?

Transformer::Transformer {
  ...
  for (auto iter : dec_layer_vec) {
    Variable *cache_k = new Variable("cache_k");
    Variable *cache_v = new Variable("cache_v");
    std::tuple<Variable *, Variable *, Variable *> dec_outs =
        (*iter)(dec_emb, total_enc_kv, pad_mask, cache_k, cache_v);
    dec_emb = std::get<0>(dec_outs);
    Variable *cache_k_out = std::get<1>(dec_outs);
    Variable *cache_v_out = std::get<2>(dec_outs);

    cache_k->set_ancestor(total_cache_k, cache_size * dec_layer_idx);
    cache_v->set_ancestor(total_cache_v, cache_size * dec_layer_idx);
    dec_layer_idx++;
  }

https://github.com/bytedance/lightseq/blob/2b5592fa658a39a914a5036e665647084d777903/lightseq/csrc/models/transformer.cu#L135

LinearOp::forward is getting cublashandle before checking if the context is built.

problem : LinearOp::forward is getting cublashandle without checking if context is built. LinearOp::backward is checking if the context is built before getting cublashandle.

solution: Modify LinearOp::forward to check if context is built before getting cublashandle.

I ran into the following two problems when using lightseq3.0.

1. I pass --path model1:model2 to ensemble model1 and model2 for generation just like fairseq-generate:

lightseq-generate $DATA_PATH \
    --path part_1/checkpoint_4_267500.pt:part_1/checkpoint_4_265000.pt \
    --batch-size 4 --beam 4 --remove-bpe \
    --gen-subset ${name} \
    --source-lang en \
    --target-lang zh \
    --max-len-a 1 \
    --max-len-b 50 \
    --lenpen 0.6 --fp16

but the operation fails in the middle with the following error(the checkpoints are from the same model).

Could you please suggest an example of ensemble?

2. When I use lightseq-generate for generation, I found that 10GB of memory is required to load a transformer_big model for lightseq while 2GB of memory is only required to load the same model for fairseq. I wonder if this is as expected?

This is loading a lightseq transformer_big model:

This is loading a fairseq transformer_big model:

Environment

• Python 3.7
• pytorch 1.12
• fairseq 0.10.2
• lightseq 3.0