1010

Ripgrep Configuration for Font Development

chris@sourcefoundry.org (Chris Simpkins) — Fri, 08 Dec 2023 21:10:01 -0500

Ripgrep is a command line file search tool known for its speed on large codebases. Here’s an approach to defining custom filtered ripgrep search types on Adobe feature, designspace, glyphs, glyphspackage, and UFO font development source files.

Create ripgrep Configuration File

Create a ripgrep configuration file on the path ~/.ripgreprc and enter the following definitions:

# Custom types
--type-add=dspace:*.designspace
--type-add=fea:*.fea
--type-add=glyphs:*.glyphs
--type-add=gpack:*.{plist,glyph}
--type-add=plist:*.plist
--type-add=ufo:*.{glyf,plist,designspace,fea}

This configuration creates six new search types, automatically filtering ripgrep searches on the defined file extensions. Our new types are:

dspace - *.designspace files
fea - *.fea files
glyphs - *.glyphs files
gpack - glyphspackage file types: *.plist, *.glyph
plist - *.plist files
ufo - UFO file types: *.designspace, *.fea, *.glyf, & *.plist files

If you prefer a different set of type names, modify the name after the equal sign and before the colon on each line of the ripgrep configuration file.

Export ripgrep Configuration Path Environment Variable

Export the configuration file path in an environment variable. Add the following line to your shell configuration rc file:

export RIPGREP_CONFIG_PATH=~/.ripgreprc

and run source on the shell configuration file. My zsh shell configuration file is on the path ~/.zshrc, and I run it like this:

source ~/.zshrc

Modify the shell configuration file path with other shells (e.g., bash). Refer to your shell documentation for details.

The above source step is not necessary with new terminal launches. You should only need to use it if your terminal is open at the time that you edit your shell configuration file.

Use the Custom Filtered Search Types

Ripgrep searches recursively by default. Navigate to the root level of your source repository before you run the commands below. Use the custom search types with the -t or --type command line option. I’ll use the short option in the examples below. Replace it with --type=[TYPE] syntax if you prefer the long option.

Search Adobe feature files

rg -t fea [SEARCH TERM]

Search designspace files

rg -t dspace [SEARCH TERM]

Search Glyphs files

rg -t glyphs [SEARCH TERM]

Search glyphspackage files

rg -t gpack [SEARCH TERM]

Search plist files

rg -t plist [SEARCH TERM]

Search UFO files

rg -t ufo [SEARCH TERM]

You may use these custom types with other ripgrep command line options. Refer to the ripgrep documentation, command line help (rg --help), and man page (man rg) documentation for the complete set of search features to customize the simple examples above.

How to Serve Brotli and GZIP Compressed Content on GitLab Pages

chris@sourcefoundry.org (Chris Simpkins) — Mon, 27 Nov 2023 00:22:14 -0500

Text content compression reduces the size of HTTP response body payloads and improves web page load times. GZIP and Brotli are both widely supported in current era web clients and use of these text compression technologies is a Lighthouse Performance Audit criterion.

GitLab Pages doesn’t serve compressed HTML, CSS, or Javascript files by default, but it is fast and straightforward to implement static GZIP and Brotli compression as part of your build and deployment workflow. After you generate the compressed files, the GitLab Pages servers will deliver the appropriate file formats to clients based on support. The approach described below should work with all static site generators. The examples are based on this site’s Hugo build workflow.

The GZIP and Brotli Compressed File Paths

This file compression approach happens during the static site build workflow. Your gzip- (extension .gz) and brotli- (extension .br) compressed files must be located in the same directory as the pre-compressed versions of the text files. Here’s an example site directory tree from the GitLab Pages documentation. It assumes your production distribution files are located in a top-level public directory.

public/
├─┬ index.html
│ | index.html.br
│ └ index.html.gz
│
├── css/
│   └─┬ main.css
│     | main.css.br
│     └ main.css.gz
│
└── js/
    └─┬ main.js
      | main.js.br
      └ main.js.gz

Compress Files in Your GitLab Pages Build

gzip is installed by default in the GitLab CI/CD runner environment. We’ll configure the brotli installation and compress all HTML, CSS, and JS files with a few lines in the .gitlab-ci.yml configuration file. This site is built with Hugo, and the example below shows the hugo static site build step before the file compression takes place. Replace the hugo build command with your static site generator build command(s) before you compress the files if you use something else.

pages:
  script:
    - apk add --update brotli
    - hugo # replace with your static site generator build command if not using Hugo
    - find public -type f -regex '.*\.\(htm\|html\|txt\|text\|js\|css\)$' -exec gzip -f -k {} \;
    - find public -type f -regex '.*\.\(htm\|html\|txt\|text\|js\|css\)$' -exec brotli -f -k {} \;

The final two configuration lines above find all HTML, CSS, and JS files in the public directory by file extensions, execute gzip and brotli compression, and write the compressed files with the appropriate file extensions in the same directory as the pre-compressed file. Change public in these commands to your production distribution directory path if you use something else.

See my .gitlab-ci.yml build and distribution configuration file for a working example of the complete Hugo site build configuration that is used to compile and deploy this site.

That’s all there is to it. Once the files are in your production environment, web clients with GZIP and Brotli support should begin to receive the compressed versions. This takes place without any further configuration from you.

Confirm Compressed Files in HTTP GET Responses with curl

Here is a curl-based one-liner that reports the Content-Encoding header on HTTP GET requests for files on your site. It can be configured with any Accept-Encoding Content Coding header value to examine the response body encoding format for assets on your site.

The following commands send an HTTP GET request—configured with an Accept-Encoding header—for this site’s home page HTML file , pipe it through head, and report out all of the content-* response headers with a grep regular expression match. Inspect the content-encoding line for the compression format. There will be no content-encoding response header when the file is not compressed. The content-length value is the response body size in bytes.

Uncompressed File

curl -H "Accept-Encoding: *" -i https://1010.gitlab.io/ 2>/dev/null | head | grep -E "content\-.*:"

Output:

content-type: text/html; charset=utf-8
content-length: 9279

gzip Compressed File

curl -H "Accept-Encoding: gzip" -i https://1010.gitlab.io/ 2>/dev/null | head | grep -E "content\-.*:"

Output:

content-encoding: gzip
content-type: text/html; charset=utf-8
content-length: 3187

brotli Compressed File

curl -H "Accept-Encoding: br" -i https://1010.gitlab.io/ 2>/dev/null | head | grep -E "content\-.*:"

Output:

content-encoding: br
content-type: text/html; charset=utf-8
content-length: 2638

You should be able to change the above URLs to any of your compressed file paths and test the content encoding in the response.

Hat tip to Mattias Geniar for the curl and head pipeline bits of the one-liners above in Test Gzip Compression of Site via Curl.

What about GitHub Pages?

GitHub Pages gzip compresses by default. No need to do anything. It just works. Despite user requests for brotli compression dating back to at least 2019, there has yet to be brotli support on GitHub Pages. This response from one of GitHub’s Open Source Community Managers indicates that support is in the queue, but it hasn’t launched to date.

Customize Mastodon to Change Your Post Character Limit

chris@sourcefoundry.org (Chris Simpkins) — Wed, 21 Dec 2022 22:26:41 -0500

New: Updated for Mastodon v4.2.1!

The Mastodon service has a 500-character limit on post text. This article documents how you change the character limit in the web user interface and API endpoint that returns instance configuration data to clients (the latter is how many mobile apps that support post length customization define your instance character limit).

These edits must be performed by an instance administrator with access to the server hosting the Mastodon software. This is an instance-wide setting that requires direct edits to the Mastodon software files and applies to all users on an instance. It is not currently possible for users to customize the default value defined on an instance.

I confirmed the instructions below function on my instance as of the Mastodon v4.2.1 release.

Customization Targets

The character limit is defined in two Mastodon source files. We will edit the following JavaScript and Ruby sources:

Mastodon < v4.2.0: compose_form.js
Mastodon v4.2.0 and later: compose_form.jsx
status_length_validator.rb

Note that the JavaScript file extension changed as of the Mastodon v4.2.0 release. The required edits did not.

Documentation of additional instance_serializer.rb file changes for earlier REST API v1 instance endpoint support are available; however, this does not appear to be relevant as of the latest API v1 and API v2 updates (at least as of the changes that landed in this pull request). I can confirm the current versions of the official Mastodon app on iOS and Android, Ivory on iOS, Toot! on iOS, IceCubes on iOS, and Tusky on Android release support the character limit increase approach described here as of the Mastodon v4.2.1 release. Please review the documentation in the link above if you need to support an earlier version of the REST API. And ping the client app development team about the API update if they use an older API version!

Prep

Log in to your Mastodon user

ssh into your server and log in to your Mastodon user. Assuming that you defined the user as mastodon at instance setup, the command is:

su - mastodon

(Optional) Create a new git branch for your modifications

Please note: Upstream/downstream git repository management is well beyond the scope of the information presented here. Please look into this issue in more detail if you are unaware of the consequences of your edits and intend to update your instance’s Mastodon version in the future.

If you cloned the Mastodon source repository with git as documented in the official instance setup docs, I recommend creating a new git branch for your modifications. This is optional but likely to help you pull new changes from the upstream Mastodon repository main branch to your local upstream tracking branch as new releases are available.

Change to the Mastodon live directory (this is the git repository root) and enter the following command:

git checkout -b mods

This generates a local modifications branch named mods starting at your instance’s Mastodon release version git tag commit.

To get back to the main branch that is tracking upstream Mastodon, either git stash or commit the changes that you make based on the documentation below, and then use the following command:

git checkout main

You can remove all live edits defined in your mods branch using the instructions below by switching back to the main branch with the command above, checking out a desired Mastodon release tag, recompiling Mastodon, and restarting your Mastodon services. The recompile and service restart commands are documented below.

Source Edits

compose_form.js(x)

Open the compose_form.js (pre-v4.2.0) or compose_form.jsx (post-v4.2.0) JavaScript file with your preferred text editor. Here, I’ll use vim with the source file path as of v4.2.1:

vim live/app/javascript/mastodon/features/compose/components/compose_form.jsx

Replace 500 in the following lines (as of Mastodon v4.2.1) with your new limit value:

return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 500 || (isOnlyWhitespace && !anyMedia));

<CharacterCounter max={500} text={this.getFulltextForCharacterCounting()} />

And write the changes to disk.

status_length_validator.rb

Open the status_length_validator.rb Ruby file with your text editor. Again, I’ll use vim with the source file path:

vim live/app/validators/status_length_validator.rb

Replace 500 in the MAX_CHARS definition below (as of Mastodon v4.2.1) with your new limit value:

class StatusLengthValidator < ActiveModel::Validator
   MAX_CHARS = 500
   URL_PLACEHOLDER_CHARS = 23
   URL_PLACEHOLDER = 'x' * 23

And write the changes to disk.

Git Patches

v4.0

Here is a git patch with the full set of v4.0.x changes:

diff --git a/app/javascript/mastodon/features/compose/components/compose_form.js b/app/javascript/mastodon/features/compose/components/compose_form.js
index 6a65f44da..b1e364451 100644
--- a/app/javascript/mastodon/features/compose/components/compose_form.js
+++ b/app/javascript/mastodon/features/compose/components/compose_form.js
@@ -90,7 +90,7 @@ class ComposeForm extends ImmutablePureComponent {
     const fulltext = this.getFulltextForCharacterCounting();
     const isOnlyWhitespace = fulltext.length !== 0 && fulltext.trim().length === 0;

-    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 500 || (isOnlyWhitespace && !anyMedia));
+    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 5000 || (isOnlyWhitespace && !anyMedia));
   }

   handleSubmit = (e) => {
@@ -277,7 +277,7 @@ class ComposeForm extends ImmutablePureComponent {
           </div>

           <div className='character-counter__wrapper'>
-            <CharacterCounter max={500} text={this.getFulltextForCharacterCounting()} />
+            <CharacterCounter max={5000} text={this.getFulltextForCharacterCounting()} />
           </div>
         </div>

diff --git a/app/validators/status_length_validator.rb b/app/validators/status_length_validator.rb
index e107912b7..a3cbe5123 100644
--- a/app/validators/status_length_validator.rb
+++ b/app/validators/status_length_validator.rb
@@ -1,7 +1,7 @@
 # frozen_string_literal: true

 class StatusLengthValidator < ActiveModel::Validator
-  MAX_CHARS = 500
+  MAX_CHARS = 5000
   URL_PLACEHOLDER_CHARS = 23
   URL_PLACEHOLDER = 'x' * 23

v4.1

Here is a git patch with the full set of v4.1.x changes:


diff --git a/app/javascript/mastodon/features/compose/components/compose_form.js b/app/javascript/mastodon/features/compose/components/compose_form.js
index e641d59f4..9b6f84fa1 100644
--- a/app/javascript/mastodon/features/compose/components/compose_form.js
+++ b/app/javascript/mastodon/features/compose/components/compose_form.js
@@ -90,7 +90,7 @@ class ComposeForm extends ImmutablePureComponent {
     const fulltext = this.getFulltextForCharacterCounting();
     const isOnlyWhitespace = fulltext.length !== 0 && fulltext.trim().length === 0;

-    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 500 || (isOnlyWhitespace && !anyMedia));
+    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 5000 || (isOnlyWhitespace && !anyMedia));
   };

   handleSubmit = (e) => {
@@ -280,7 +280,7 @@ class ComposeForm extends ImmutablePureComponent {
           </div>

           <div className='character-counter__wrapper'>
-            <CharacterCounter max={500} text={this.getFulltextForCharacterCounting()} />
+            <CharacterCounter max={5000} text={this.getFulltextForCharacterCounting()} />
           </div>
         </div>

diff --git a/app/validators/status_length_validator.rb b/app/validators/status_length_validator.rb
index e107912b7..a3cbe5123 100644
--- a/app/validators/status_length_validator.rb
+++ b/app/validators/status_length_validator.rb
@@ -1,7 +1,7 @@
 # frozen_string_literal: true

 class StatusLengthValidator < ActiveModel::Validator
-  MAX_CHARS = 500
+  MAX_CHARS = 5000
   URL_PLACEHOLDER_CHARS = 23
   URL_PLACEHOLDER = 'x' * 23

v4.2

Here is a git patch with the full set of v4.2.x changes:

diff --git a/app/javascript/mastodon/features/compose/components/compose_form.jsx b/app/javascript/mastodon/features/compose/components/compose_form.jsx
index 9222b2dc8..962310a28 100644
--- a/app/javascript/mastodon/features/compose/components/compose_form.jsx
+++ b/app/javascript/mastodon/features/compose/components/compose_form.jsx
@@ -100,7 +100,7 @@ class ComposeForm extends ImmutablePureComponent {
     const fulltext = this.getFulltextForCharacterCounting();
     const isOnlyWhitespace = fulltext.length !== 0 && fulltext.trim().length === 0;

-    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 500 || (isOnlyWhitespace && !anyMedia));
+    return !(isSubmitting || isUploading || isChangingUpload || length(fulltext) > 5000 || (isOnlyWhitespace && !anyMedia));
   };

   handleSubmit = (e) => {
@@ -297,7 +297,7 @@ class ComposeForm extends ImmutablePureComponent {
             </div>

             <div className='character-counter__wrapper'>
-              <CharacterCounter max={500} text={this.getFulltextForCharacterCounting()} />
+              <CharacterCounter max={5000} text={this.getFulltextForCharacterCounting()} />
             </div>
           </div>
         </div>
diff --git a/app/validators/status_length_validator.rb b/app/validators/status_length_validator.rb
index dc841ded3..9cb1ec94b 100644
--- a/app/validators/status_length_validator.rb
+++ b/app/validators/status_length_validator.rb
@@ -1,7 +1,7 @@
 # frozen_string_literal: true

 class StatusLengthValidator < ActiveModel::Validator
-  MAX_CHARS = 500
+  MAX_CHARS = 5000
   URL_PLACEHOLDER_CHARS = 23
   URL_PLACEHOLDER = 'x' * 23

Rebuild Mastodon

Change to the live sub-directory from your mastodon user home directory:

cd live

and re-compile your Mastodon project with the following command:

RAILS_ENV=production bundle exec rails assets:precompile

Restart Your Mastodon Services

Exit to your root user:

exit

and execute the following commands to restart (or reload) your Mastodon services:

systemctl restart mastodon-sidekiq
systemctl reload mastodon-web

Reload your Mastodon instance in the web browser UI, and you should be ready to post with the new character limit.

Confirm the Mastodon REST API Update

REST API clients will rely on the updated max character count data in your instance API endpoint.

Here is an example of the relevant part of the JSON response from the mastodon.social instance:

"configuration": {
    "urls": {
      "streaming": "wss://mastodon.social"
    },
    "accounts": {
      "max_featured_tags": 10
    },
    "statuses": {
      "max_characters": 500,
      "max_media_attachments": 4,
      "characters_reserved_per_url": 23
    },

By default, the max_characters field is defined with 500 characters.

Your instance’s response should be different. Use a GET request to retrieve the JSON data at the following URL:

https://[ENTER YOUR DOMAIN HERE]/api/v2/instance

You should see the following updates in the response:

"configuration": {
    "urls": {
      "streaming": "wss://[YOUR DOMAIN]"
    },
    "accounts": {
      "max_featured_tags": 10
    },
    "statuses": {
      "max_characters": 5000,
      "max_media_attachments": 4,
      "characters_reserved_per_url": 23
    },

max_characters will list the custom character limit value you defined in the steps here if all went well with your patch. You can use this API check to troubleshoot any issues you encounter in clients that support custom character limits.

Close and reload all REST API clients that support custom post lengths to apply the update. This includes Mastodon applications on mobile devices. You should now see the updated character count value in the UI.

UI Update Examples

Mastodon Web UI

Mastodon official app on iOS

Tusky on Android

Edits

2023-10-13: updated documentation with the requirements in Mastodon v4.1.x and v4.2.x, including the compose_form.jsx path change as of v4.2.0.

2022-12-22: added a “Git Patch” section with a Mastodon v4.0.2 git patch that describes the source code changes.

1010

Ripgrep Configuration for Font Development

Create ripgrep Configuration File

Export ripgrep Configuration Path Environment Variable

Use the Custom Filtered Search Types

Search Adobe feature files

Search designspace files

Search Glyphs files

Search glyphspackage files

Search plist files

Search UFO files

How to Serve Brotli and GZIP Compressed Content on GitLab Pages

The GZIP and Brotli Compressed File Paths

Compress Files in Your GitLab Pages Build

Confirm Compressed Files in HTTP GET Responses with curl

Uncompressed File

gzip Compressed File

brotli Compressed File

What about GitHub Pages?

Further Reading

Customize Mastodon to Change Your Post Character Limit

Customization Targets

Prep

Log in to your Mastodon user

(Optional) Create a new git branch for your modifications

Source Edits

compose_form.js(x)

status_length_validator.rb

Git Patches

v4.0

v4.1

v4.2

Rebuild Mastodon

Restart Your Mastodon Services

Confirm the Mastodon REST API Update

UI Update Examples

Mastodon Web UI

Mastodon official app on iOS

Tusky on Android

Edits