Advanced features

1. Semgrep: Flex Mode!

This tutorial covers some of Semgrep's newest features, and is mostly geared toward experienced Semgrep rule authors.

We'll start by reviewing an example from the composition tutorial that introduced metavariable-regex.

This time we have pre-filled the correct answer to get you started.

SEMGREP RULE

rules:
  - id: use-decimalfield-for-money
    patterns:
    - pattern-inside: |
        class $M(...):
          ...
    - pattern: $F = django.db.models.FloatField(...)
    - metavariable-regex:
        metavariable: '$F'
        regex: '.*(price|fee|salary).*'
    message: Found a FloatField used for variable $F. Use DecimalField for currency fields to avoid float-rounding errors.
    languages: [python]
    severity: ERROR

TEST CODE

from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=64)
    description = models.TextField()
    # this is ok
    price = models.DecimalField(max_digits=6, decimal_places=2)
    # this is also ok
    return_rate = models.FloatField()
    # Semgrep finds this because old_fee ends in the word fee, which is in the regex above
    old_fee = models.FloatField()
    # match this
    price_inc = models.FloatField()

ANSWER + EXPLAIN

Without the metavariable-regex here, the metavariable $F would match on line 9 as well.

However, you can use metavariable-regex to restrict metavariable matches based on the variable names.

You can also use regular expressions directly in semgrep patterns. See how in the next example! 🚂

2. Regex in patterns

SEMGREP RULE

Sometimes, regular expressions are the best tool for the job.

Semgrep lets you combine all of its other tools with regular expressions so you can use each where it makes most sense.

This example currently tries to use pattern-regex to match baseURL variables that do not start with http:// or https:// but it doesn't quite work.

Try using pattern-regex's cousin, pattern-not-regex to express this more elegantly.

rules:
  - id: invalid-base-url
    message:
      The 'baseURL' is invalid. This may cause links to not work if deployed. Include the scheme (e.g., https://).
    patterns:
      - pattern: baseURL = "..."
      - pattern-regex: (?!.*http).*
    languages: [generic]

TEST CODE

# ruleid: invalid-base-url
baseURL = "example.com"
baseURL = "unfortunateurlwithhttp.com"
baseURL = "http://anotherurl.com"
baseURL = "https://anotherurl.com"
DefaultContentLanguage = "en"
Paginate = 10
enableRobotsTXT = true

ANSWER + EXPLAIN

rules:
  - id: invalid-base-url
    message:
      The 'baseURL' is invalid. This may cause links to not work if deployed. Include the scheme (e.g., https://).
    patterns:
      - pattern: baseURL = "..."
      - pattern-not-regex: https?://.*
    languages: [generic]

Regex matching is not the only form of logic that can be built into a Semgrep pattern.

Keep going for math comparisons like $X > 0! (Fun fact: 0! = 1)

3. Metavariable comparisons

Just as metavariable-regex could be used to restrict metavariables based on a regex, metavariable-comparison can be used to restrict metavariables based on mathematical comparisons.

This example currently matches on RSA keys that are initialized with 2048 bits or fewer.

Update it to match the message, so it only alarms on RSA keys with fewer than 2048 bits.

SEMGREP RULE

rules:
  - id: use-of-weak-rsa-key
    message: RSA keys should be at least 2048 bits based on NIST recommendation.
    patterns:
      - pattern: |
          KeyPairGenerator $KEY = $G.getInstance("RSA");
          ...
          $KEY.initialize($BITS);
      - metavariable-comparison:
          comparison: $BITS <= 2048
          metavariable: $BITS
    languages: [java]
    severity: WARNING

TEST CODE

import java.security.KeyPairGenerator;

public class WeakRSA {

  static void rsaWeak() {
    // ruleid: use-of-weak-rsa-key
    KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
    keyGen.initialize(512);
  }

  static void rsaOK() {
    // ok: use-of-weak-rsa-key
    KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
    keyGen.initialize(2048);
  }
}

ANSWER + EXPLAIN

rules:
  - id: use-of-weak-rsa-key
    message: RSA keys should be at least 2048 bits based on NIST recommendation.
    patterns:
      - pattern: |
          KeyPairGenerator $KEY = $G.getInstance("RSA");
          ...
          $KEY.initialize($BITS);
      - metavariable-comparison:
          comparison: $BITS < 2048
          metavariable: $BITS
    languages: [java]
    severity: WARNING

4. Metavariable pattern

Do you remember back in the composition tutorial when we first introduced metavariable-regex and noted how it let you write code about your code about your code?

Metavariable-pattern is like that, but maybe one layer deeper. Or maybe infinite layers deeper. We sort of lost track.

Take a good hard look at this example to figure out what it is doing. There is a pattern-not embedded in the logic for a specific metavariable. This one was too hard to turn into a puzzle, so we are just giving it to you. Enjoy!

SEMGREP RULE

rules:
  - id: disallow-old-tls-versions2
    message: Detected an old tls version
    patterns:
      - pattern: |
          $CONST = require('crypto');
          ...
          $OPTIONS = $OPTS;
          ...
          https.createServer($OPTIONS, ...);
      - metavariable-pattern:
          metavariable: $OPTS
          patterns:
            - pattern-not: >
                {secureOptions: $CONST.SSL_OP_NO_SSLv2 | $CONST.SSL_OP_NO_SSLv3
                | $CONST.SSL_OP_NO_TLSv1}
    languages:
      - javascript
    severity: WARNING

TEST CODE

function bad() {
  // ruleid:disallow-old-tls-versions2
  var constants = require('crypto');
  var sslOptions = {
  key: fs.readFileSync('/etc/ssl/private/private.key'),
  secureProtocol: 'SSLv23_server_method',
  secureOptions: constants.SSL_OP_NO_SSLv2 | constants.SSL_OP_NO_SSLv3
  };
  https.createServer(sslOptions);
}

5. Everybody loves Autofix

Everybody loves Raymond autofix because it doesn't just tell you how to fix your code - it does it for you!

Right now, autofix is mostly limited to single-line patterns and single-line fixes, but it still comes in quite handy.

Semgrep's rule format supports a fix: key that supports metavariable replacement, much like message fields.

In this example try writing a fix: that matches the message description.

SEMGREP RULE

rules:
  - id: use-sys-exit
    pattern: exit($X)
    message: |
      Use "sys.exit" over the python shell "exit" built-in. "exit" is not available on all Python implementations.
    languages: [python]
    severity: WARNING

TEST CODE

import sys

if False:
    # ok
    sys.exit(2)

if True:
    # ruleid: use-sys-exit
    exit(3)

ANSWER + EXPLAIN

rules:
  - id: use-sys-exit
    pattern: exit($X)
    message: |
      Use "sys.exit" over the python shell "exit" built-in. "exit" is not available on all Python implementations.
    languages: [python]
    severity: WARNING
    fix: sys.exit($X)

6. Go forth and find things!

You've reached the end of this tutorial. Semgrep has many other features that you can read about at the docs.

Here's a favorite pattern that shows off what Semgrep can do. This rule searches for code injection in Django apps using eval() and has found real vulnerabilities!

SEMGREP RULE

rules:
  - id: user-eval
    patterns:
    - pattern-inside: |
        def $F(...):
          ...
    - pattern-either:
      - pattern: eval(..., request.$W.get(...), ...)
      - pattern: |
          $V = request.$W.get(...)
          ...
          eval(..., $V, ...)
      - pattern: eval(..., request.$W(...), ...)
      - pattern: |
          $V = request.$W(...)
          ...
          eval(..., $V, ...)
      - pattern: eval(..., request.$W[...], ...)
      - pattern: |
          $V = request.$W[...]
          ...
          eval(..., $V, ...)

TEST CODE

import django
from textwrap import dedent

def unsafe(request):
    # ruleid: user-eval
    code = request.POST.get('code')
    print("something")
    eval(code)

def unsafe_inline(request):
    # ruleid: user-eval
    eval(request.GET.get('code'))

def unsafe_dict(request):
    # ruleid: user-eval
    eval(request.POST['code'])

def safe(request):
    # ok
    code = """
    print('hello')
    """
    eval(dedent(code))