Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Last update: Mar 15, 2022

Overview

FeatureHasher

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Note, this requires Jina>=2.2.4.

Example

Here I use FeatureHasher to hash each sentence of Pride and Prejudice into a 128-dim vector, and then use .match to find top-K similar sentences.

from jina import Document, DocumentArray, Flow

# load 
   
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').convert_uri_to_text()

# cut into non-empty sentences store in a DA
da = DocumentArray(Document(text=s.strip()) for s in d.text.split('\n') if s.strip())

# use FeatureHasher in a Flow
f = Flow().add(uses='jinahub://FeatureHasher')

embed_da = DocumentArray()
with f:
    f.post('/', da, on_done=lambda req: embed_da.extend(req.docs), show_progress=True)

print('self-matching...')
embed_da.match(embed_da, exclude_self=True, limit=5, normalization=(1, 0))
print('total sentences: ', len(embed_da))
for d in embed_da:
    print(d.text)
    for m in d.matches:
        print(m.scores['cosine'], m.text)
    input()

           [email protected][I]:🎉 Flow is ready to use!
	🔗 Protocol: 		GRPC
	🏠 Local access:	0.0.0.0:52628
	🔒 Private network:	192.168.178.31:52628
	🌐 Public address:	217.70.138.123:52628
⠹       DONE ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 100% ETA: 0 seconds 40 steps done in 1 second
total sentences:  12153
The Project Gutenberg eBook of Pride and Prejudice, by Jane Austen

   
     *** END OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

    
      *** START OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

     
       production, promotion and distribution of Project Gutenberg-tm

      
        Pride and Prejudice

       
         By Jane Austen This eBook is for the use of anyone anywhere in the United States and 
        
          This eBook is for the use of anyone anywhere in the United States and 
         
           by the awkwardness of the application, and at length wholly 
          
            Elizabeth passed the chief of the night in her sister’s room, and 
           
             the happiest memories in the world. Nothing of the past was 
            
              charities and charitable donations in all 50 states of the United

In practice, you can implement matching and storing via an indexer inside Flow. This example is only for demo purpose so any non-feature hashing related ops are implemented outside the Flow to avoid distraction.

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

Owner

Jina AI

python写的一款免杀工具（shellcode加载器）BypassAV，国内杀软全过（windows denfend）

This repository detects a system vulnerable to CVE-2022-21907 and protects against this vulnerability if desired

A Docker based LDAP RCE exploit demo for CVE-2021-44228 Log4Shell

log4j2 dos exploit,CVE-2021-45105 exploit,Denial of Service poc

RedTeam-Security - In this repo you will get the information of Red Team Security related links

Searches through git repositories for high entropy strings and secrets, digging deep into commit history

Vulnerability Exploitation Code Collection Repository

Jolokia Exploitation Toolkit (JET) helps exploitation of exposed jolokia endpoints.

Fetch Chrome, Firefox, WiFi password and system info

PwdGen is a Python Tkinter tool for generating secure 16 digit passwords.

Password list generator for password spraying - prebaked with goodies

Format SSSD Raw Kerberos Payloads into CCACHE files for use on Windows systems

This a simple tool XSS Detection Suite for CTFs games

A simple Log4Shell Scan with python

Remote control your Greenbone Vulnerability Manager (GVM)

Exploit-CVE-2021-21086

Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage and potential attacks.

hackinsta: a program to hack instagram

Pre-Auth Blind NoSQL Injection leading to Remote Code Execution in Rocket Chat 3.12.1

macOS persistence tool