VidiNLP: NLP Library for Linguists

As a linguist and researcher, I’ve spent countless hours analyzing text data using popular Python libraries like spaCy and NLTK. While these tools are powerful and efficient, I found myself repeatedly writing the same boilerplate code for common NLP tasks across different projects. This repetition was taking valuable time away from the actual linguistic analysis I wanted to focus on.

This frustration led me to create VidiNLP (named after my nickname, Vidi), a high-level NLP library that wraps multiple underlying NLP tools into a single, intuitive interface. Let me share why I built it and how it might help you in your own text analysis workflow.

The Problem: Too Much Boilerplate

Consider a simple task like extracting n-grams from text. Using spaCy directly, you might write something like this:

import spacy
from collections import Counter

def get_ngrams(text, n=2, top_n=None, lowercase=True, ignore_punct=True):
    nlp = spacy.load("en_core_web_sm")
    doc = nlp(text)
    
    # Process tokens based on parameters
    tokens = [
        token.text.lower() if lowercase else token.text
        for token in doc 
        if not (ignore_punct and token.is_punct)
    ]
    
    # Generate n-grams
    ngrams = [" ".join(tokens[i:i+n]) for i in range(len(tokens) - n + 1)]
    
    # Get top_n most common n-grams if specified
    if top_n:
        ngrams_counter = Counter(ngrams)
        return ngrams_counter.most_common(top_n)
    
    return ngrams

That’s quite a bit of code for a relatively simple operation. And this is just one of many text processing functions you might need in a typical NLP project.

The Solution: VidiNLP

With VidiNLP, the same task becomes refreshingly concise:

from vidinlp import VidiNLP

nlp = VidiNLP()
ngrams = nlp.get_ngrams("The quick brown fox jumps over the lazy dog", n=2, top_n=3, lowercase=True, ignore_punct=True)
print(ngrams)

Why Use VidiNLP?

1. Unified Interface

One of the main advantages of VidiNLP is that it provides a consistent interface across different NLP tasks. Instead of switching between libraries (spaCy for tokenization, NLTK for lemmatization, scikit-learn for TF-IDF, etc.), you can access all these functionalities through a single, coherent API.

2. Time-Saving Convenience

VidiNLP includes ready-to-use functions for common tasks like:

• Text preprocessing (tokenization, lemmatization, POS tagging)
• Text cleaning (removing stop words, punctuation, HTML, etc.)
• Sentiment and emotion analysis
• Keyword extraction
• Document similarity
• Readability analysis
• Named entity recognition
• Topic modeling

3. Advanced Linguistic Analysis

As a linguistics researcher, I’ve incorporated analyses that aren’t readily available elsewhere, particularly around text structure:

structure = nlp.analyze_text_structure(text)
print(structure)

This provides detailed metrics like:

• Sentence and paragraph length statistics
• Discourse marker identification
• Pronoun reference ratios
• Parts-of-speech distribution
• Sentence type distribution (simple, compound, complex)

4. Integration of Multiple Approaches

For tasks like sentiment analysis, VidiNLP combines multiple approaches to provide a more comprehensive result. This means you don’t need to choose between different implementations—you get the best of each.

When to Use VidiNLP

There is a trade-off to consider: VidiNLP requires installing several dependencies (spaCy, Pandas, scikit-learn, vaderSentiment), which might be overkill if you only need a couple of simple NLP functions. If your project only requires basic functionality from one or two libraries, you might be better off using those directly.

However, for research projects requiring diverse text analysis capabilities, VidiNLP can significantly reduce development time and code complexity. It’s particularly valuable when you need:

1. A wide range of NLP functions
2. Detailed linguistic analysis
3. A consistent interface across different NLP tasks
4. Quick prototyping without writing boilerplate code

Getting Started

Installation is straightforward:

pip install vidinlp
python -m spacy download en_core_web_sm

And then you can immediately start analyzing text:

from vidinlp import VidiNLP

# Initialize the analyzer
nlp = VidiNLP()

# Clean text with various filters
cleaned = nlp.clean_text(
    "Hello! This is a test 123... <p> with HTML </p>",
    remove_stop_words=True,
    remove_none_alpha=True,
    remove_punctuations=True,
    remove_numbers=True,
    remove_html=True
)

# Get sentiment scores
sentiment = nlp.analyze_sentiment("This movie was absolutely fantastic!")

# Extract keywords using TF-IDF
keywords = nlp.extract_keywords("Machine learning is a subset of artificial intelligence", top_n=3)

Conclusion

VidiNLP was born out of my own need as a linguistics researcher to make text analysis workflows easier. By wrapping powerful NLP libraries into a single, intuitive interface, it lets researchers focus on analyzing their data rather than writing boilerplate code.

Whether you’re conducting academic research, performing text analysis for business insights, or just exploring a corpus of text, VidiNLP aims to make your work more efficient and your code more readable.

The library is continually evolving with new features and improvements. I welcome feedback from the NLP and linguistics research communities to make VidiNLP even more useful for real-world text analysis tasks.