Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Last update: Mar 15, 2022

Overview

FeatureHasher

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Note, this requires Jina>=2.2.4.

Example

Here I use FeatureHasher to hash each sentence of Pride and Prejudice into a 128-dim vector, and then use .match to find top-K similar sentences.

from jina import Document, DocumentArray, Flow

# load 
   
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').convert_uri_to_text()

# cut into non-empty sentences store in a DA
da = DocumentArray(Document(text=s.strip()) for s in d.text.split('\n') if s.strip())

# use FeatureHasher in a Flow
f = Flow().add(uses='jinahub://FeatureHasher')

embed_da = DocumentArray()
with f:
    f.post('/', da, on_done=lambda req: embed_da.extend(req.docs), show_progress=True)

print('self-matching...')
embed_da.match(embed_da, exclude_self=True, limit=5, normalization=(1, 0))
print('total sentences: ', len(embed_da))
for d in embed_da:
    print(d.text)
    for m in d.matches:
        print(m.scores['cosine'], m.text)
    input()

           [email protected][I]:🎉 Flow is ready to use!
	🔗 Protocol: 		GRPC
	🏠 Local access:	0.0.0.0:52628
	🔒 Private network:	192.168.178.31:52628
	🌐 Public address:	217.70.138.123:52628
⠹       DONE ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 100% ETA: 0 seconds 40 steps done in 1 second
total sentences:  12153
The Project Gutenberg eBook of Pride and Prejudice, by Jane Austen

   
     *** END OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

    
      *** START OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

     
       production, promotion and distribution of Project Gutenberg-tm

      
        Pride and Prejudice

       
         By Jane Austen This eBook is for the use of anyone anywhere in the United States and 
        
          This eBook is for the use of anyone anywhere in the United States and 
         
           by the awkwardness of the application, and at length wholly 
          
            Elizabeth passed the chief of the night in her sister’s room, and 
           
             the happiest memories in the world. Nothing of the past was 
            
              charities and charitable donations in all 50 states of the United

In practice, you can implement matching and storing via an indexer inside Flow. This example is only for demo purpose so any non-feature hashing related ops are implemented outside the Flow to avoid distraction.

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

Owner

Jina AI

xp_CAPTCHA(白嫖版) burp 验证码识别 burp插件

S2-061 的payload，以及对应简单的PoC/Exp

test application for the licence key web app.

Phishing-Crack tools to punish friends

This repository uses a mixture of numbers, alphabets, and other symbols found on the computer keyboard

A small Minecraft server to help players detect vulnerability to the Log4Shell exploit 🐚

Password-Manager GUI

AttractionFinder - 2022 State Qualified FBLA Attraction Finder Application

Exploiting CVE-2021-42278 and CVE-2021-42287

The Linux defender anti-virus software ported to work on CentOS Linux.

Open source vulnerability DB and triage service.

Web-eyes - OSINT tools for website research

Small Python library that adds password hashing methods to ORM objects

Update of uncaptcha2 from 2019

Tools for investigating Log4j CVE-2021-44228

Cloud One Container Security Runtime Events Forwarder

RedDrop is a quick and easy web server for capturing and processing encoded and encrypted payloads and tar archives.

Fast and customizable vulnerability scanner For JIRA written in Python

ssh-audit is a tool for ssh server & client configuration auditing.

This is a proof-of-concept exploit for Grafana's Unauthorized Arbitrary File Read Vulnerability (CVE-2021-43798).

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

Owner

Jina AI

xp_CAPTCHA(白嫖版) burp 验证码 识别 burp插件

S2-061 的payload，以及对应简单的PoC/Exp

test application for the licence key web app.

Phishing-Crack tools to punish friends

This repository uses a mixture of numbers, alphabets, and other symbols found on the computer keyboard

A small Minecraft server to help players detect vulnerability to the Log4Shell exploit 🐚

Password-Manager GUI

AttractionFinder - 2022 State Qualified FBLA Attraction Finder Application

Exploiting CVE-2021-42278 and CVE-2021-42287

The Linux defender anti-virus software ported to work on CentOS Linux.

Open source vulnerability DB and triage service.

Web-eyes - OSINT tools for website research

Small Python library that adds password hashing methods to ORM objects

Update of uncaptcha2 from 2019

Tools for investigating Log4j CVE-2021-44228

Cloud One Container Security Runtime Events Forwarder

RedDrop is a quick and easy web server for capturing and processing encoded and encrypted payloads and tar archives.

Fast and customizable vulnerability scanner For JIRA written in Python

ssh-audit is a tool for ssh server & client configuration auditing.

This is a proof-of-concept exploit for Grafana's Unauthorized Arbitrary File Read Vulnerability (CVE-2021-43798).

xp_CAPTCHA(白嫖版) burp 验证码识别 burp插件