Categories
Uncategorized

Monitoring for email delivery problems

This post describes how I monitor for mail delivery issues, in particular mails where delivery is blocked by recipients policy, on a small server running Debian Bookworm, with Postfix and the monitoring-plugins-check-logfiles package.

If the system you are delivering to is RFC compliant, i.e. follows the normal internet rules, then if it does not accept an email from you due to policy (such as denying email from your IP address), it should return a status code which starts with 5.7 This will show up in your /var/log/mail.log as a status containing

'dsn=5.7'

I run an icinga2 system to monitor systems for errors, and whenever I discover a problem which had not been picked up by monitoring, and it was one which could have been detected earlier, I don’t consider the issue resolved until I have added a check to look for it in future. For this particular case I use check_logfiles – which can be found in the Debian package monitoring-plugins-check-logfiles, installed on the outgoing mail server, with these configuration items.

Mail server with /var/log/mail.log

Here are the items I added to a mail server running postfix with logs going to /var/log/mail.log, with other items already being monitored.

/etc/nagios/nrpe.d/check_logfiles_dsn57.cfg
command[check_logfiles_dsn57]=/usr/bin/sudo /usr/local/bin/check_logfiles_dsn57
/etc/sudoers.d/nagios_check_logfiles_dsn57
nagios ALL=(root) NOPASSWD:/usr/local/bin/check_logfiles_dsn57
/usr/local/bin/check_logfiles_dsn57
#/bin/sh

/usr/lib/nagios/plugins/check_logfiles --tag=dsn57 --criticalpattern="dsn=5.7" --logfile=/var/log/mail.log

Mail server with journald

Transitioning away from traditional syslog, with postfix logs in journald needs a slight tweak to the monitoring check_logfiles, which I patched as shown in the patch attached to Debian bug #1060859. I also changed the local script to be

/usr/local/bin/check_logfiles_dsn57
#/bin/sh

/usr/lib/nagios/plugins/check_logfiles_journal_identifier --tag=dsn57 --type=journald:identifier='postfix/smtp' --criticalpattern="dsn=5.7"

Icinga2 server configuration

The service check for check_logfiles_dsn57 is likely not to find a new adminstratively denied mail the next time it is run (I have a check_interval of 1h), so I use the following non standard parameters for the nrpe service which does the check

  check_command = "nrpe"
  check_interval = 1h
  retry_interval = 24h
  volatile = true
  vars.nrpe_command = "check_logfiles_dsn57"

The ‘retry_interval = 24h’ leaves the critical alert for the problem visible until it has been investigated, and will then be cleared by re-running the test.

I have ‘nagstamon‘ (from the Debian package) running on my desktop as a constant overview which needs less screen space than the icingaweb page, which provides more information for a detailed investigation.

Categories
Uncategorized

Setting up a Small Pleroma Server

This is a work in progress and will be updated as I get time. Initially it simply documented what I did for my own server, but now it reflects a build on a test server. I intend to refine it iteratively.

My first installation of the Pleroma server social.paladyn.org, was through a Yunohost server, which I had set up on a QEMU Virtual Machine, which runs Debian Bullseye. This worked quite well to try, but had several drawbacks, including that it shared the same external IPv4 address with several other systems, which introduced complexities.

Pleroma, Mastodon or what ?

Mastodon is the best known name in the ActivityPub Federated area, and my first venture into the Fediverse was as @JohnLines@mstdn.io – but I wanted to understand the platform well enough that I could contribute to making it more widely available – probably as part of a Small Organisation Server.

Debian has a team working on providing Social Networking for Developers, and another working on packaging. Through the Social team I am @jlines@pleroma.debian.social which steered me towards Pleroma, particularly as Pleroma and Mastodon work together.

That working together, but not being Mastodon, is important as for a protocol to be an IETF Standard it must have two independent, interoperating implementations. Diversity in implementing the Fediverse makes for a healthier ecosystem.

Yunohost

Yunohost is an all-in-one solution, which is well worth investigating if you have a system which is not already set up doing other things, but in my case I decided to migrate to running Pleroma directly on a Debian Bullseye system which was already running Nginx, but did not have much running which would conflict with Pleroma.

Pleroma from Source

The official Pleroma instructions for install on Debian have two methods, one is what they call an OTP release – where they supply a pre-built binary image for many popular computer architectures, the other is to build it from source.

As my end goal is a Debian package, I went down the build from source route. In the following instructions I have adhered to the Pleroma instructions, rather than Debian standards for file locations etc.

Picking a name and setting up DNS.

You will need a name for your Federated Social Media System. Although this document is about installing Pleroma you may want to consider a more generic name, for example setting up social.example.com, in the same way as you might use www.example.com, rather than apache.example.com to allow a future switch to nginx, or vice versa. If you set up the name pointing to your target system now then there is more chance for it to propagate before you need it. You will also be asked an admin email address later, and an address for sending email notifications. You may need to do some preparation for these too.

Preparing for the installation.

In order to make sure the target system knows the dependencies which should be installed I have started a Debian package called pleroma-installer.

pleroma-installer

Note that in these commands I have a blank line between each command. They may wrap round when shown in your browser, but should be entered as one line. I have put parts you will need to change for your system in italics.

At the moment this simply asks for dependencies, and can be downloaded from https://paladyn.org/john/tmp/pleroma-installer_0.0.1_all.deb – install it by

wget -nc https://paladyn.org/john/tmp/pleroma-installer_0.0.1_all.deb

sudo apt install  ./pleroma-installer_0.0.1_all.deb

This will install the dependencies, and apt will know that they were installed for a reason, and not try to clean them up.

The plan is to create an apt repository for this installer package to enable it to be updated by apt, however as it is an installer, not a real package this is not a high priority.

Create a user and directories

On installation the package will create a pleroma user and a directory /opt/pleroma which is the main place the files are kept. If /opt/pleroma exists it will be moved to /opt/pleroma.yyyymmddhhmm and the pleroma files will be fetched again.

There will be a warning about Federation not working until it is configured.

At present this is as far as the installer package goes, partly as the next section will have some questions to answer.

Generate the instance

cd /opt/pleroma

sudo -Hu pleroma MIX_ENV=prod mix pleroma.instance gen 

Say Y to install rebar3. You will be asked for the name and email addresses you chose above. Otherwise I used the defaults, except for answering ‘y’ to Do you want to store the configuration in the database.

The installation should complete, saying ‘All files successfully written!’

sudo -Hu pleroma mv config/{generated_config.exs,prod.secret.exs}

sudo -Hu postgres psql -f config/setup_db.psql

The above steps put the configuration where pleroma expects it, and do the initial database setup. As I chose the option to store the configuration in the database I did:

sudo -Hu pleroma MIX_ENV=prod mix ecto.migrate

You should see logs about lots of files being compiled.

At this stage pleroma should be installed and ready to run, start it with

sudo -Hu pleroma MIX_ENV=prod mix phx.server

You will end up with pleroma running as a background process.

Set up nginx

If you started with an empty server then the installer should have pulled nginx, but if you already had apache2 installed then it will not switch web servers on you. I have not tested pleroma behind apache2. but there is a sample apache configuration in /opt/pleroma/installation

Check if nginx is running by

systemctl status nginx

and stop it by

systemctl stop nginx

Now install an SSL certificate for your new subdomain – note that it will need to exist by this stage.

sudo certbot certonly --email myuser@example.org -d social.example.org --standalone

This can fail if some other program is using port 80 (the non SSL secured web port on your computer) – for example if or apache is running, of if your domain is mistyped, or does not exist yet.

sudo cp /opt/pleroma/installation/pleroma.nginx /etc/nginx/sites-available/pleroma.nginx

sudo ln -s /etc/nginx/sites-available/pleroma.nginx /etc/nginx/sites-enabled/pleroma.nginx

Edit /etc/ngix/sites-available/pleroma.nginx to change all occurrences of sample.tld to your server name e.g. social.example.com

sudo cp /opt/pleroma/installation/pleroma.service pleroma.service

sudo systemctl enable --now pleroma.service

You should now be able to access your server at https://social.example.com/ – check here if this does not work.

Create your first user

sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new myuser myuser@example.com --admin

Part of the output from the pleroma.user command will be a password reset URL – enter the link into a web browser, set your password and you should have set up a pleroma server !

Feedback on this post is welcome, either by comments on this post (note that they are moderated so will not appear instantly) or via federated social media to @jlines@pleroma.debian.social.

Troubleshooting

Pleroma has quite a number of pieces, and there are stages where the setup may run into problems. Here are some of the ones which people have encountered, together with what to do about them.

Certbot unable to set up the SSL certificate

Nginx (or apache) showing the home page, not Pleroma

If you went to the web page of the site you set up, for example https://social.example.org/ – but see a default web page then web server configuration has not taken notice of the new site you added. The first thing to do is restart nginx

systemctl restart nginx

and look again, if you see the home page still then you will need to look further.

The sites which are live can be found in /etc/ngnix/sites-enabled/ – as symbolic links to the real configurations.

You can check by

ls -l /etc/nginx/sites-enabled

You should see a line like

lrwxrwxrwx 1 root root 40 Nov 3 18:39 pleroma.nginx -> /etc/nginx/sites-available/pleroma.nginx

Nginx not running

Removing pleroma

When there is a real Debian package this should just a a matter of

apt remove pleroma

but at present it does not have an uninstall script, and what it should do to tidy up is not obvious.

Removing the database

You might want to do this in order to do a re-install cleanly

systemctl stop pleroma.service

sudo -Hu postgres psql -c 'DROP DATABASE pleroma; '

sudo -Hu postgres psql -c 'DROP USER pleroma; '

Removing the files

All the files are created under /opt/pleroma, but if you run the installer a second time it moves /opt/pleroma in order to have a clean install, but not lose anything if you want to go back, or compare configuration files.

Categories
Uncategorized

Funding the Fediverse

For anything continue to operate, it has to be, in some form, financially viable. (I regard economics as one of the key decision making tools available to people – see How do we decide ?). I am also suspicious of systems which try to conceal their economic model. (see Transparancy and Trust, and Who pays for WhatsApp).

Monolithic Social Media

Western commercial social media, by which I mean systems such as Facebook, YouTube, TikTok, WhatsApp etc has two primary funding streams.

  1. Selling advertising space on their own platform
  2. Selling information about their users.

As some users are starting to find the advertisements when using the platform intrusive, they are developing a new stream, where users pay to use the platform without seeing any advertising, but this does not interfere with stream 2, the gathering of data about their user base.

The Chinese Social Networking system WeChat also has an eCommerce system, WeChat Pay, which gives it another revenue stream, an area in which Meta group and Alphabet have struggled to establish a foothold.

Federated Social Media

Federated social media systems have the following funding options:

Donations from users

This is the model which funds, in theory, many Federated social media sites. Generally they are free to use, and the system relies on some people being sufficiently generous to keep the site in operation. The sites are generally run by someone with a good technical understanding. The users, prior to joining the site, did not have a particular relationship with the person/group who runs it, although it may be set up for some general class of users, for example those with an interest in the environment.

Funding by Governments and large companies

The main present example of this is the European Commission. For a government or large organisation with a Public Relations department, or similar, funding a Federated site is a trivial expense and has the potential to extend their reach, particularly as it is unlikely to be impacted by controversies related to that platform. (for example people may like what you do, but stop following you on, say Twitter – if they do not like what you do they will not follow you on any platform anyway).

Some large organisations may fund their own social media system, for example Truth Social, generally with a particular political slant.

Journals, Newspapers, TV channels

Large media organisations i.e. newspapers and television channels are a natural fit for Federated Social Media, as I point out in Federated Social Media and Journalism. They also have a role as a conduit for funding from readers (if those readers pay some form of subscription to the journal or media outlet) to the journalists. Other means of financially rewarding journalists, writers and artists exist, such as Patreon, but they too involve a middle-man. The journals add value for the reader by associating, and risking, their reputation on the quality of the writing they publish, and take a cut for that. The relationship between the size of the cut and the value added too complex to discuss here, but compared to the other costs involved the technical side of running servers is (probably) small.

Funding by small organisations and individuals.

If the technical knowledge needed to run a social media system could be reduced then the cost or running one for the benefit of members is within the reach of many small organisations. The advantage of running their own system and federating with others is that the content can be more focused. By funding directly, without advertising, there is less risk that the social media feed will end up carrying content which does not match the goals of that organisation.

Selling Advertising

Although nobody, as far as I know, is doing this yet, it should be possible to support a social media site through advertising sold through a broker, without needing that broker (which in many ways is what Facebook etc are) needing to own the site outright. Being federated users could post content onto those sites, which would be visible to their followers on other sites, who would also see some advertisements. Google, Facebook, and Amazon already act as advertising brokers and have mechanisms to place advertising on third party web sites. It might require and extension to the ActivityPub protocol to insert the advertising posts.

Sponsorship

People who make a living being an influencer could run their own site, and, clothing and lifestyle companies, who currently sponsor them could do so directly.

Finance, Ownership and Control.

For all forms of mass media there is a complex, but important, relationship between the sources of revenue which pay for it, who owns it, and who can control it. For example while WeChat is owned by Tencent and TikTok is owned by ByteDance, in both cases the Chinese government has (according to some people) a substantial degree of control over their activities.

The control of Federated Social Media systems is different, in that the name of the system, for example PeerTube – a Federated Video sharing system, and individual ‘instances’ have their own ownership and agendas. There are Peertube systems which host videos about technical subjects, such as https://peertube.debian.social/ and Framasoft, and big creator of Free Federated Software, both funded by donations.

You could think of, say Twitter, as an unfederated Social Media system, which provokes thought about the balance between who pays for it (a mixture of advertisers and investor(s)) and who controls it.

Categories
Works in progress

The Future – Feudal or Federal

This is a work in progress, incomplete but published on the Release Early, Release Often principle. Feedback is very much welcomed.

Back in 2012 Bruce Schneier, author of Applied Cryptography and respected security expert, wrote a set of articles on Feudal Security, and The Battle for Power on The Internet, describing how the trend towards channeling all our Internet activity (which is becoming most of our communicating, shopping and learning) through a small number of giant internet companies. He compared this to the way that vassals in medieval times swore allegiance to the barons, who in return mostly provided a degree of protection, but often abused that power.

He said that it was time for governments to establish regulatory frameworks to control the corporations. Unfortunately, in many respects the situation has become worse. Many Internet Giants have revenue greater than the GDP of most countries, so most governments are negotiating from a position of weakness.

The concept that Feudal is the only possible way means that, for example, people wishing to leave WhatsApp due to some scandal or dislike of its policies or market domination, tend to switch, for example to Telegram i.e. leaving one closed system for another.

I like the video explaining the concept of Federated Systems at https://framatube.org/w/9dRFC6Ya11NCVeYKn8ZhiD – particularly as it is on PeerTube – a Federated Video Sharing system.

I hope an Internet of Federated systems can provide almost the same functionality and convenience that the giant monolithic systems can, through the organisations they already belong to.

To make this possible the Free Software community needs to make sure that these alternative systems are readily available to non technical people.

Categories
Uncategorized

SPF Problems

The Sender Policy Framework (SPF) is an important part of preventing email forgery on the Internet, preventing spammers from forging mail which pretends to be from you.

SPF records

It is common in modern mail systems for the email for an organisation to be delivered by some specialist third party, rather than the organisation itself. The SPF (Sender Policy Framework) record is a way of declaring to the world who you trust to be sending mail on your behalf.

Getting this right is important, as it is rather similar to a Power of Attorney, you are telling the world that they should trust the systems listed as if they were you when they receive an email. It is essentially an anti-forgery system.

If your SPF record is incorrect you are likely to encounter mail delivery problems. The impact will vary depending on the recipient, but the best way to fix them is to make sure your SPF record is correct using an online SPF checker.

SPF checkers

Several companies provide a web page which allow you to enter your domain name and they will tell you if there are problems with your SPF record – and offer to sell you as solution if a problem is detected. They all tend to tell you roughly the same if your SPF record is correct, but differ in how informative the message is if your SPF is invalid.

A search for ‘spf checker’ will turn up more. You can check any domain with them, not just your own, so you can see if a mail problem from some domain is SPF related.

SPF problem – too many DNS lookups

As the DNS is so important to the whole internet, RFC 7208 (one of the ‘rules of the road’ of the Internet) states that an SPF record MUST not require than 10 DNS lookups. Without this there would be a way for a bad person to attack people on the internet in a way which is difficult to trace – an example of this is explained at SPF Too Many DNS Lookups, in the section ‘Why is There an SPF Lookup Limit?’. That link also contains some general suggestions as to how to tidy the SPF record.

SPF and Surveymonkey

SurveyMonkey, a popular survey management company, can sometimes have its SPF record incorrectly added to that of customers, but they themselves state, in their help page, that ‘You do not need to add SPF or DKIM records to your domain when using SurveyMonkey.

SPF and bulk mail sending companies

You might have a contract with a bulk mail sending company, for example to send out a newsletter. Depending on the company you may be asked to add their SPF list to yours, and their SPF list may be quite large. Not all mailers require this, so it is worth checking.

Another possibility to consider is setting up a subdomain, such as newsletter.example.com, or a more generic emails.example.com, and asking the bulk mailing company to use that. That subdomain will then have its own SPF list, which will not normally need to include your own email provider.

SPF and changing email providers

It is quite natural, if you change email providers, to add the new one to your SPF list, but sometimes the step of removing the previous one to your SPF list can be forgotten.

But mail still goes to Gmail !

Just as the laws which most people follow do not apply to everyone, for example if they have diplomatic immunity, Google can choose which rules they follow. As in the case of Harry Dunn, just because they can does not make it correct.

If a wicked person was to divert a major road’s worth of traffic down your residential street it would cause chaos, but Google effectively owns multiple motorways, so they are immune to the problems which affect others.

Categories
Uncategorized

DNS and Identity

Many organisations are very concerned about their identity, hiring expensive consultants to redesign their logo, with the result being reviewed at the most senior level. Their identity on the Internet tends to receive much less attention, being delegated, without much thought to some ‘techie’ people, either inside or outside the organisation, without much guidance as to what the online identity should be.

At the highest level identities, or Names on the Internet are divided into a number of Domains in a structure a bit like a tree, although conventionally pictures with the root at the top! From the root come a number of major branches, the .com, .edu, .org, (etc) naming areas, originally for Commercial, Educational (American), other Organistations, and .uk, .de, .fr for United Kingdom, German (Deutschland), France and so on. Each of these branches is controlled by some naming authority, and some of those authorities sell, through brokers, names in that space on the open market. Others names are not for sale, for example you can not buy a name which ends .nhs.uk, which is reserved for the British National Health Service.

Most individuals, and some small organisations will not even start down this road and will exist purely as an identity within one of the Internet giants, for example Gmail, Facebook, Twitter etc – in which case their identity is whatever works within the rules of that system.

This article is for people and organisations who have, or are planning to have their own identity on the Internet. In examples I will assume you are using example.com as your name.

The Domain Name System (DNS)

Email

The mail system depends on the DNS, so if the DNS is set up incorrectly then there will be mail delivery problems.

MX records

(These are essential to mail delivery, and I will update this article to cover them)

SPF records

It is common in modern mail systems for the email for an organisation to be delivered by some specialist third party, rather than the organisation itself. The SPF (Sender Policy Framework) record is a way of declaring to the world who you trust to be sending mail on your behalf.

Problems with your SPF record will cause problems delivering your email, and have their own article on SPF Problems.

Categories
Uncategorized Works in progress

Producers, Consumers and Intermediaries

This is a work in progress, published in this state, but which needs substantial revision to finish it.

In the real world the things we want as consumers are often supplied by a chain, where some form of intermediary sits between us and the ultimate producer of whatever that thing was, for example

Farm or Factory -> Retailer -> Consumer

Writer -> Publisher -> Reader

Our place in this chain will very according to what we are doing, when a writer buys food, they switch from a producer to a consumer. The ability to specialise accounts for much of human progress, and the intermediary also has an important role and should add value to the process.

Problems in the system tend to come down to abuses of power, usually because one element of the chain has some form of monopoly which prevents, for example shoppers from going to an alternative shop to buy some product, but, for example aggressive negotiations by supermarkets may force farmers into being unable to sell their products are a sustainable price.

Risk and reward

The intermediary is sometimes taking a risk, for example a shop purchases goods, and sells them at a profit, but the profit has to cover the goods which the shop buys and then is unable to sell.

Problems arise if, for any stage in the process, the profits are either excessive in relation to the risk, or fail to cover the risks. In free market theory, competition should prevent this from happening, for example if a shop is making excessive profits, then a competitor will notice an easy profit to be made and move in to undercut the original, profiteering shop.

If this is prevented by some form of monopoly then the system breaks down. Again, in theory, there are markets build around a natural monopoly – for example the supply of utility services to houses. Although an element of competition can be introduced, for example by creating a market in gas intermediaries, the pipework carrying the gas has to be run by a body which is controlled by regulation rather than competition.

Licensing and Copyright management

Publishers, Record Companies, YouTube, CCLI potentially add value by collecting money from people who watch media or listen to music and the artists and musicians who create it.

In the case of YouTube, in general for their advertising supported model, they are not taking money directly, but receiving money from advertisers

The situation becomes complicated in the case of intermediaries (collective rights management) who are not really in the chain, for example PRS for Music – as they do not actually know whose content is actually being used. At least the Public Lending Right tries to match library loans to author remuneration.

For all creative content a major issue is ensuring that revenue generated from that content is distributed appropriately down the chain, from the purchaser through the intermediary they deal with, for example Amazon for Kindle books, then possibly other intermediaries, such as a publisher, and finally the author(s). For a paper book, bought from a high street bookseller, they will have costs, such at rent and rates to pay, which an online seller will not have at the same level, so the purchaser should expect to pay more. It would help someone trying to make ethical decisions on any kind of creative content to have more transparency on where their money is going.

Categories
Small Organisation Server

Small Organisation Server – the Target

The end goal of the Small Organisation Server project is a system which can provide everything needed for a small organisation, such as the Ambridge Garden Club, to have a presence on the Internet, with transparent accounting for how it is funded. The organisation should be able to own its data and systems. Although it will have to pay for hosting, as it should – there is no such thing as a free lunch – that hosting should be on a standard platform which does not lock the organisation into any particular vendor.

Simplicity of Administration

Small organisations should not need to have more than a minimal knowledge of computer administration.

Automatic updates

The software should update with the minimum attention needed from the administrator, who should not need to deal with a different update system for each component.

Modular

Not every small organisation will want every facility, for example a choir might want some form of music repository, and a garden club might not have a use for that.

Simple to use

Single identity and password

Federation and other organisations

Many people have multiple interests, and the system should not try to be the only one they use.

Stability

There is a clash between a desire to have the latest and greatest, and having a system which focuses on continuity. Although the organisation will be interested in keeping up to date with what it is interested in, e.g. gardening, that does not mean that it should need the very latest software.

Non-goals

All designs involve compromises, trade offs between, for example complexity and power. These are the choices I believe to be appropriate for the target types of Small Organisation.

Extreme privacy and security.

Hosting on a base you do not own, such as a hosted virtual computer (or even a real computer in a data centre) requires a level of trust in your host. There is a theoretical possibility, for example, that your hosting provider can read all of your data. The same applies to government level agencies, well funded criminal gangs etc. However such operations are expensive, and if that is a concern then you are in a different category of organisation.

Scaling to enormous sizes

Some software or systems do not scale to support thousands, or millions of people on the same platform. This causes them to be dismissed by people who want to be able to run huge systems. If you are expecting to grow to those sort of sizes you should be planning for some paid dedicated IT staff.

Applications – what can it do ?

The purpose of a Server is to serve, to serve the needs of the people who own it (Also true for the servers which belong to Facebook, Twitter, Google, Amazon etc). Here I put some of the things which might be useful to a Small Organisation on the Internet.

Mail

One of the key building blocks of the internet, and also often now used as a proxy for identity. It should be possible for members of the organisation to contact each other through email, and possibly at their choice have their emails forwarded to another system, or read to write emails from the server. This should be compliant with technical standards for mail authentication, such as DMARC, without needing the administrators to have to understand the technicalities. Ideally it should support filtering via Sieve, and reading and writing mail either through a web interface or standard email client programs.

Mailing lists

Sending emails to large numbers of recipients is a specialist area, particularly if there are doubts about whether they want the emails or not, and if the small organisation wants to do that type of thing they should pay a specialist. However for cases like contacting committee members, or possibly, for example a small charity to update previous donors (who are willing to be contacted) a mailing list manager can be useful.

Web pages

These are the public face of the organisation, and it should be relatively easy for those members of the organisation who represent it to update them. This will probably not be every member, and it should be clear, at least internally, who said what. Some form of content management system, such as WordPress or Drupal, integrated with the rest of the user administration should be possible.

Surveys

Some form of Survey software may prove useful, but it does not have to be Google Forms, or Microsoft Forms or SurveyMonkey – it may be well be that LimeSurvey will do the job.

Videos

If the organisation creates video content they should be able to decide whether to pay to host it, and allow others to view it without advertising, or to have someone else host it for ‘free’ paid for by advertising, which may not align with their ideals.

Note that, probably, this is not a large a risk as it might seem. Although a Temperance Society might in theory find it’s videos interrupted by advertisements for alcoholic beverages this would not be an effective use of the advertiser’s spend.

It is still an area where groups should have more choice about whether to host their own video content, and being able to run Peertube on their own server would allow this. This also allows the ability to publish content only available to members, should they want, and to live stream meetings.

Chat/Instant messaging

There are a number of options for this, depending on requirements. Some of them keep a permanent record of everything which has been said, such as Matrix, which might be wanted for important decisions, but overkill for general chat. The established standard in this area is XMPP, a Federated chat protocol which was the basis for Facebook Messenger and Google Talk before they decided to only allow it be used within their own communities. It can, but does not have too, keep a record of chats on the server.

Social media (microblogs)

Some organisations may want to produce small snippets of information, similar to Tweets, intended for public consumption. They may also want their members to be able to do this, but may want to distinguish between an official view and member’s opinions. Software based around the ActivityPub standard works for this.

Social networking

Some groups may want members to be able to create small amounts of content primarily for sharing with friends, for which something like Diaspora may be more appropriate.

Calendars and events

People will probably want to know when the next meeting is and similar needs. Organisations can host their own calendars and do not need to Google or Microsoft to host them, which means that they do not have to require their members to have Google or Microsoft accounts.

These calendars can be added to smartphones or integrated into calendar programs on a PC, such as Thunderbird or Outlook, which saves members from having to add them manually.

Video conferencing/virtual meetings

Organisations should be able, if they wish, to run meetings and conferencing on systems they own and operate themselves, using a system such as Jitsi, or BigBlueButton (more suitable for a larger organisation).

Status

At present all of the components exist, and someone with an interest in computers can put forward such a system, as has been done at least in part for Debian and Wikipedia, but there is a scarcity of information making it simple for someone without computer skills to put such a system together.

Systems like FreedomBox have a similar aim, but targeted towards individuals hosting on their own hardware. Yunohost is also based on Debian, and is the closest I can find to a Small Organisation Server. The aspect I don’t know about is it’s stability across Debian upgrades. It’s installation instructions are still based on Debian version 10, whereas Debian’s stable release is now version 11, and I know the upgrade was fairly painless. Their Use Cases for NGOs is quite similar to this post.

Categories
Uncategorized

Federated Social Media and Journalism

Although journalists take a keen interest in Commercial Social Media (CSM), primarily meaning Facebook (and its subsidiaries, WhatsApp, Instagram etc), and Twitter (and possibly Tik Tok, and YouTube) there are particular reasons why journalists should be wary of them, and be aware of, and use and promote Federated Social Media alternatives.

Commercial Social Media is a rival, not a friend

Traditional Media (newspapers and broadcasters) and Commercial Social Media compete for advertisers, and CSM does so more successfully, as it can deliver a more accurate profile of its readership to the advertisers, who are the people who are paying for it.

CSM does not employ any journalists, benefiting from the work of journalists paid by Traditional Media, while undermining the revenue which pays their salaries. A career as an ‘Influencer‘ is probably not what most serious journalists aspire to.

Commercial Social Media will inevitably become political

As Commercial Social Media becomes more accurate in its profiling of its user base it will know more and more about the return on investment it can deliver to an advertiser from any particular user. There will come a point where the most valuable thing that many social media users can provide in return for free hosting of their posts and pictures will be their votes.

By pushing their readership towards CSM, when they use Facebook and Twitter as their means of relating to their user base, Traditional Media organisations, are indirectly encouraging this trend. This is particularly ironic as the idea of interference in democratic elections is abhorrent to most journalists in the free world. (it is interesting that the expression free world is taken to mean free as in freedom, and journalists do not confuse it with free as in free newspapers).

As a case in point, The Guardian article on ‘David Puttnam hits out at government as he quits House of Lords” has Share buttons for Facebook, Twitter (and email), but The Guardian does not have an independent social media presence of its own, despite the one of the significant points of the article being the government’s lacklustre response to the report on ‘Digital Technology and the Resurrection of Trust‘. (This 153 page report touches on many of issues of transparency and trust)

What are the alternatives

These are just suggestions which journalists may find worth investigating. Journalism being the production and distribution of reports on current events based on facts and supported with proof or evidence, the ways that information flows in today’s world seems worth studying.

The common factor in all of the following suggestions is that they are Federated, so a Traditional Media organisation can set up their own presence (or instance) in these systems and not lock their readership/viewers in. They would all be able to be used in a subscriber model, in the same way as a newspaper operating behind a paywall if this turned out to be the best business model.

They are also all Free Software, so the cost of experimenting and learning about them is low, and they are supported by enthusiastic and helpful communities. When setting up such systems it is worth considering what they should be called. For experimental purposes any domain name is suitable, but in production they should be subdomains of the main internet presence to inherit its trust – see It is good to be a tree for why improving understanding of trust is becoming increasingly important in the online world.

Diaspora

as an alternative to Facebook. I do not yet have a Diaspora account, though it looks worth investigating.

XMPP

as an alternative to WhatsApp (see Who pays for WhatsApp). This is an Internet Standard Instant Messaging protocol, with at least two good choices of server systems (the part that might be run by a newspaper) and a wide range of clients (the part the used by their readership), including web clients so readers do not need to install a particular app if they do not want to.

I am on xmpp as jlines@debian.org

As XMPP is its own protocol there is no clash in using names similar to the email addresses already in use. If for example The Guardian were to offer an XMPP service to its subscribers (a potential way to delivery added value to the subscription at low cost the the newspaper), they could use addresses of the form fred.bloggs@subscribers.theguardian.com to distinguish them from staff.

ActivityPub (Mastodon or Pleroma)

as an alternative to Twitter. This works best for content which is intended to be publicly shared, and can be boosted (a bit like being retweeted) across multiple instances.

I am on the Debian instance of pleroma as @jlines@pleroma.debian.social.

Peertube

as an alternative to YouTube. The organisation running the server pays the hosting and network bandwidth costs to host their own streaming media, but by peering with other instances allows the other instances viewers, to see the other instances content, some of it using hosting organisation’s bandwidth, and vice versa.

The Dark Side

The ability to operate a Federated Media instance to publish content using their own rules about what is permitted does mean that it can become a platform for views or content which has been blocked or banned from CSM, for example (or so I hear – I have not looked at it myself) Gab.

I do know there are some instances which are echo chambers for conspiracy theorists, but distinguishing truth from falsehood, fact from fiction and conspiracy from cover up should be the essence of journalism, and equipping the public to do the same should be part of the mission.

On the other hand, operating a Federated Media Instance in their own domain (as a subdomain of the domain where they have built up a web presence) would allow traditional journalism publishers to leverage the trust in their existing ‘brand’.

hope on the horizon

Better tools for investigating trustworthiness of information found on the internet are always interesting, and a project called EUNOMIA looked interesting. It is not yet at the stage where it is useful to the average social media user, but people with in interest in journalism, politics or sociology might benefit from being aware of it, and – in example of the way things change, its domain now points to some form of gambling site, so I have removed the link to its live demo. It is described here, and source code on Gitlab.

Categories
Uncategorized

User level backup with borgmatic

I have been using Bareos (previously I used Bacula) for many years, both at work and at home. Since Freedombox uses Borg, and it is always good to understand what is available I have been looking at Borg for comparison. This led me to Borgmatic, as a good way to setup and manage Borg backups. The Debian package gave me a smooth route to system backups (the systemd files required to automate the running need to be manually installed and configured, but this is not too tricky – the main thing to watch out for is adjusting the path to executable in /etc/systemd/system/borgmatic.service)

I wanted to see if I could run a separate backup of key files from my user account so they could be held on another server as an extra level of security. I discovered a problem, in that some, but not all, of those files are really held on an NFS server, and root on my desktop does not have access to them. This prompted me to see if I could run borgmatic, as installed at a system level from the Debian bullseye package, as a normal user. It turns out to be possible, and as it might be useful to others I am documenting it here.

mkdir .config/borgmatic
generate-borgmatic-config -d .config/borgmatic/config.yaml

Note above that borgmatic uses -d for the name of the configuration file when it is being generated, but -c elsewhere.

Now edit the borgmatic configuration to pick up the files to be backed up and the place they should be stored. This could be a remote borg server, completely different from the one used for system backups.

validate-borgmatic-config -c .config/borgmatic/config.yaml

borgmatic -c .config/borgmatic/config.yaml init --encryption repokey
borgmatic -c .config/borgmatic/config.yaml --verbosity 1 --files

This should check the configuration, and backup the files.

Now to run it as a systemd user service with a timer.

mkdir -p .config/systemd/user

within this directory create these files – based on the system ones

borgmatic.timer

[Unit]
Description=User borgmatic backup

[Timer]
# Backup at some time when the system will be, but not busy
OnCalendar=17:15
Persistent=true

[Install]
WantedBy=timers.target

borgmatic.service

[Unit]
Description=borgmatic user john backup
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
Restart=no
LogRateLimitIntervalSec=0

ExecStart=/usr/bin/borgmatic -c /home/john/.config/borgmatic/config.yaml --verbosity -1 --syslog-verbosity 1

Now run the commands

systemctl --user daemon-reload
systemctl --user enable borgmatic.timer --now

These should now run your backup once per day at the time you specified. Note that this assumes your user is logged on, which is the case for me – getting into user timers which run when the user is logged out is another issue – and requires root access, which none of the preceding steps need (as long as the borgmatic package is installed.