Tuesday 7 February 2023

Restoring NextCloud to a New OpenSuse build

I have been running my Nextcloud instance on OpenSuse 15.3 which has reached EOL, so it was time to upgrade. Unfortunately, something went awry with the in-place 15.3->15.4 upgrade resulting in a trashed btrfs filesystem. Fair enough, not a problem, I have tiered backups using rsnapshot on a separate volume which include not only the Nextcloud www and data directories but also /etc for good measure. Before each rsnapshot run, I also mysqldump a copy of the Nextcloud database into the top of the data directory.

This time, unlike previous OpenSuse+Nextcloud builds, I'm not going to install the official Nextcloud package since it's always an older verion. I can just plonk the backup copy into /srv/www/htdocs and be done with it. I chose the LAMP server pattern for my OpenSuse install, restore the backup directory, and reload the database into MariaDB. All very simple but it ain't working, so what have I missed?

  1. Set permissions on the restored files. They should be owned by wwwrun and be chmod 750
  2. Apache config (you can use the Yast sysconfig tool for this),
    1. Add ssl, rewrite and headers to the modules list in sysconfig
    2. Make sure Apache is started with SSL option
    3. Restore the /etc/apache2 config directory from backup so it has vhosts and SSL config correct 
  3. PHP config (I'm still with PHP 7 at this point)
    1. Make sure all the Nextcloud PHP module dependencies are installed
    2. Install php-apcu and php-cli
    3. Edit PHP config as outlined in my previous posting to enable opcache, apcu and up the memory limits.
  4. Cron config
    1. Restore /etc/cron.d from backup
    2. Change permissions on all the files to 644
  5. Letsencrypt config (seems to be installed be default in the 15.4 LAMP pattern)
    1. Restore /etc/letsencrypt from backup
  6. Networking
    1. Make sure hostname and IP address are set so that SSL certificates work externally
    2. Open firewall ports for http and https

If you remember these then the rebuild takes maybe an hour. A lot of this is restoring files - I'm using SSD's so if you are stuck with HDD's this might be rather longer.

Another trick is that OpenSuse 15.4 now offers a downloadable updated ISO image which includes patches. This speeds up the install a fair bit.

Sunday 22 January 2023

Running KVM VM's as a regular user on OpenSuse 15.4

 Things you need to do to allow a regular user to use virt-manager for VM's

  1. Add user to the following groups: kvm, libvirt, qemu
  2. These bits let regular users redirect USB devices to a VM...
    Add the line: org.spice-space.lowlevelusbaccess     yes:no:yes
    to: /etc/polkit-default-privs.local 
  3. After this run: /sbin/set_polkit_default_privs
  4. Add the line: usr/bin/spice-client-glib-usb-acl-helper  root:root  4755
    to: /etc/permissions.local
  5. After this run: chkstat --system --set
Credit to simon at serverfault

Tuesday 22 March 2022

Shoehorning Duallies - A Slightly Bigger Build

 


My previous compact dual cpu builds were based around the (now long discontinued) Coolermaster Elite 360 and motherboards that were more or less ATX sized - or SSI CEB which is a little deeper at 10.5 inches rather than 9.6. At 147x380x410mm (WxHxD) the case comes in at 22.9 litres in volume and had two 5.25" bays and two 3.5" bays which provided ample room for storage (and judicious routing of cables). However, those were sold off a while ago.

Looking to reduce my power consumption a bit, now that utility bills are rising, I decided to consolidate my desktop and server machines into a single unit running Linux/KVM. This would need to be quite a beefy unit but I don't want anything too big. Shopping around, I came across the Kolink KLA-002 at overclockers.co.uk which looked rather interesting. At 175x410x385mm (WxDxH) and 27.6 litres, this case is about an inch wider than the Elite 360 but can fit an EATX motherboard with some adjustment. It's also dirt cheap so I'm happy taking a drill to it.

It looks like there are two optical bays up top but the upper bay is blocked by the front panel buttons etc. moulded into the top of the front cover. You could mount an internal drive or two there but I am going to tuck the extra length of the PSU cables in there to keep things neat. 

There are two 2.5 inch mounts on the base plate in front of where an ATX motherboard would fit - however, these would be blocked by an EATX motherboard. There are screw holes at the bottom left of the case to fit either a 3.5" HDD or a 120mm fan (albeit with a fairly restrictive grill). 

There are also two 3.5" drive bays at the bottom right which will need to be removed to get a larger motherboard in. It is possible to slide a board in behind the drive bays but you risk damaging components and reaching any connectors there is impossible. 

To remove it you need to drill out 5 rivets (arrowed). Two on the bottom edge, two under the front cover and one on the base. They can easily be replaced by a screw and a nut if you want to reinstate the bays later. 

Once the bays are removed there is space for an EATX motherboard with some caveats. There are only standoffs in the ATX board positions so you will need some additional board support. You can get plastic standoffs that clip into the empty screw holes but even then the bottom right corner position falls over a hole in the backplate for cable routing. If the board is the full 330mm deep then it will come right up against the front fans which can be tricky if there are, for example, right angle SATA connectors there. In that case, consider investing in slimmer 120mm front fans.  

 
The motherboard I chose for this build was the Huananzhi X99 T8D. Huananzhi is one of the better home-grown Chinese manufacturers who tend to have reasonable VRM designs, functional BIOS's and even a degree of support. This is an unusual board which targets the small number of non-standard OEM Haswell Xeons that support DDR3 as well as DDR4. These were adopted by the likes of cloud providers back in the day so are plentiful and cheap on the second-hand market. Huananzhi have also added some more modern features - bootable M.2 slots for NVME and SATA SSD's, USB 3, onboard BIOS code indicators and onboard power and reset buttons for testing. It also has 7.1 audio and dual (Realtek) Gigabit ethernet. The board is also a fraction smaller than EATX at 290x310mm which makes fitting it into the case a breeze.

To go with this board I have two Xeon E5-2696 v3 CPU's with 18 cores each and a top turbo speed of 3.8GHz. These can be had for under £100 off Aliexpress which is ridiculous for the amount of power you get. Each of the E5's has a multicore Passmark score equivalent to an Intel Core i9-10900K. There is a BIOS mod which will allow the full turbo speed to be applied to all the CPU cores but this can strain the VRM's somewhat. This mod works for all V3 Xeons - search for "S3TurboTool" if you are interested. Usefully this motherboard also supports LRDIMM's which are also remarkably cheap since not all motherboards support them. £260 for 8x32GB DDR3-1600 LRDIMMS from the well-known auction site, which happily overclock to DDR-1866 speeds.

Getting back to the build, all the front panel connections are on the bottom edge of the board. Rather than messily routing the cables through the case, they can be bundled together and fed through the holes in the front of the case cleared by removing the drive bays. All except the USB 3 connector which is too big. This is fed through the slot under the optical drive and tucked out of the way under the bottom edge of the front fans. There is just enough room for the cables round the edge of the optical drive when the front cover is snapped into place. I've also put in some mesh filters over the fans since they are intakes. 


Here is the motherboard fitted into the case. Huananzhi provide suitable plastic standoffs for cases that don't have standoffs in the right place.

The coolers are Raijintek Aidos with the fans replaced with 90mm Arctic PWM PST ones. One annoying feature of the motherboard is that it only has 4-pin fan control for the CPU fans. The PST feature of Arctic fans allows the PWM signal to be passed on to other fans. The front CPU also controls the two 120mm intake fans - which are also Arctic PST models. The rear CPU initially also controlled the rear exhaust fans - two Arctic 80MM PST models that I found got a bit noisy as things warmed up. I ended up connecting the rear fans to one of the 3-pin fan headers with a speed reduction resistor. The fan speed vs temperature profiles for each CPU can be independently configured in the BIOS and it actually works! Unfortunately, the profiles rely on chipset temperature sensors rather than the CPU sensors so there is a bit of a lag in their response. Linux fancontrol can be used to get round this.

The two 8-pin EPS power connectors and the main ATX power connector are all at the top edge of the board so those cables can be tucked above the DVD-RW drive. I found that one of the EPS cables could be fed through the channel on the side of the 5.25" drive bay. The other could be tucked into the gap between the case and the PSU and then down the gap between the PSU and the rear fans. 

The only cables I routed behind the motherboard were the SATA data cable for the DVD-RW drive, and the SATA power for the SSD's. As I've mentioned before, I find the SuperMicro slim SATA cables to be the best for my purposes. You could also run a GPU PCIE power cable down here if needed - I chose a GTX 1650 card which doesn't require a power cable and provides enough oomph for the few games I do play, like Oolite or a bit of Minecraft.

You can also just see the PST signal cable from the front CPU fan which also connects to the two front fans back here.  

Having removed or blocked most of the storage bays in the case, I do need to have some more storage slots than just the on-board M.2 connectors. On the well-known auction site, I found a card that holds two 2.5" SATA drives and fits into a PCI slot. It doesn't take any power or signals from the slot - there are two SATA data and one SATA power connector on the card edge.

Two drives aren't really enough though, so I bought two cards, removed the bracket from one and stacked them together using 8mm spacers and 25mm screws. This gives just enough space to fit a 7mm SSD and four of them occupy a slightly-larger-than-single-slot card. Just found an "official" but bulkier version from DeLock.

 I did have to saw/file off a few of the PCI slot fingers to fit round motherboard components but since there are no connections there it is not a big deal. You can see it fully loaded at the bottom of the whole-case shot above.

I like to have one physical drive per VM using KVM raw pass-through. Easy to move to a new host and there is no storage contention when VM's are doing heavy activity. 

 So there you have it:

2x18 Haswell Xeon cores at up to 3.8 GHz

256GB 8-channel DDR3 1600 LRDIMMS overclocked to 1866

1xM.2 NVME SSD + 4x2.5" SATA SSD's

All in a 27 litre case, and costing well under a grand.           

Thursday 30 April 2020

Nextcloud Address/Calendar Bbackups

I've been looking at an easy way to mirror my Nextcloud instance to a secondary server and came across Bernie_O's calcardbackup which neatly plugs a hole. While it is pretty easy to use rsync or similar to copy the files across the nextcloud database is a bit of a problem. You could use myqsl dump but that replicates the entire installation which is not quite what I want. However, when you get down to it, the bits that really matter are calendars and address books. The rest - user lists etc. are pretty much set up once and left alone.

What the script does is dumps out users calendars and address books in .ics and .vcf format - which can easily be imported when needed. What I do (as the web user) is:
  1.  Dump out the users data a files in the nextcloud admins account (not compressing them into a package, and only keeping 14 days backups):
    /usr/bin/calcardbackup/calcardbackup "/srv/www/htdocs/nextcloud" -x -r 14 -o "/srv/www/htdocs/nextcloud/data/admin/files/Backups"
  2. Tell nextcloud to rescan the admin's files to make them visible in the nextcloud client
    php /srv/www/htdocs/nextcloud/occ files:scan -p "admin/files/Backups"
  3. Once I conformed that these worked OK I added them to the web user's cron jobs to run daily at 2am
The dumps are then rsync'ed along with everything else.

Friday 15 November 2019

Chuwi Minibook First Look

Received my Chuwi Minibook (crowdfunded on Indiegogo) and have been having a play so here are my first thoughts. For my basic setup I am more-or-less duplicating the configuration I used for my GPD Pocket with updated versions of the packages in question.

I ordered the M3-8100Y version with 16GB of RAM. However, I did not order any additional SSD since I intended to use an M.2 PCI-E MicroSD adapter for an internal drive. This means I can just take the internal MicroSDXC from my GPD Pocket rather than having to copy anything. The card also draws at most maybe 0.5W as a storage device which is a lot less than an equivalent SATA or NVME SSD. Yes, it's slow but I seriously don't notice that much since I'm not a gamer.

Some observations about the Minibook.
  1. Chuwi have configured M3-8100Y with TDP-up settings that give a minimum turbo of 1.6GHz. It would be great if this was configurable in a future BIOS with the options of regular TDP (1.1GHz) and TDP-down (800MHz) for those of us who want to extend battery life. I'll detail my experiments with Throttlestop in this area down below.
  2. It's annoying that the minimum PD power requirements is 24W (12Vx2A) rather than 18W (9Vx2A) since there are a lot of 18W adapters and power packs around. Not sure whether this might be adjustable or is hard wired. On the other hand, it makes sense considering the potential maximum power draw of the Minibook (15W peak TDP for the CPU alone).
  3. It would be nice if the default keyboard light status could be configured in the BIOS to Off/On/Last Setting for mainly for resumption from suspend/hibernate.
  4. The screen is a seriously shiny fingerprint magnet and I'm still looking for a good oleophobic matt screen protector in the UK. I ordered at AtFoliX one from eBay and it was far too big.
  5. The Intel drivers for the M3-8100Y seem to handle external displays very badly - the GPD Pocket was painless but getting scaling and orientation right with the new control panel doesn't really work.
Now, on to my Throttlestop configuration (I have yet to try XTU)...
  1. I have downvolted the CPU core by 100mV and the CPU cache by 50mV to reduce power consumption. This works for me but depends on your CPU. This works for all frequency settings.
  2. I have left the TDP limits as they were, since they can't be configured per profile in Throttlestop. Instead I use turbo frequency limits to control power consumption. I haven't fiddled with GPU settings since I don't really game that much - so its power consumtpion is generally minimal. Geekbench 4 figures are in [brackets].
  3. Profile 1 is 1.6gGHz limit for both single and dual core turbo (this is as low as it goes) and is the default I use when on battery. Power consumption seems to be around 3.5W peak (CPU only). [2207/3845]
  4. Profile 2 is 1.8/2.2GHz and with a CPU peak power of around 5W. The fan doesn't seem to spin up at this level and keeps things quiet. [2798/4807]
  5. Profile 3 is 2.2/2.7GHz, with a CPU peak power of around 7W. The fan spins up but is generally not too obtrusive. This is my default when plugged in. [3289/5706]
  6. Profile 4 is the default 2/7/3.4Ghz that the Minibook comes with. With undervolting I am seeing a peak of maybe 10W here but the fan does get noisy. [3939/6771]
I have yet to do any thermal mods so there's still more scope for noise limiting. Battery life for Profile 1 running the CPU flat out, display brightness 40% and keyboard light off is around 2.5-3 hours. Idle CPU power draw is around 0.5W so there is scope for quite good longevity under normal use.

For those of you with NVME drives that draw 4+W, default CPU settings and Windows Update/Indexing thrashing both, I'm not surprised battery life is short. Once Windows settles down things will improve a lot but a heavily used power hungry SSD could easily halve battery life.    

Sunday 27 January 2019

Nextcloud on OpenSuse 15.0 Part 2

Now we've got the latest Nextcloud (15.X) running more or less satisfactorily on OpenSuse 15.0, it's now time to do a bit more detailed configuration.

First of all we need to set up a cron job to do the regular housekeeping that nextcloud needs. Currently, it kicks off this activity whenever a page loads but as more content gets added this begins to become a bottleneck. So, in a terminal session do the following as root:
  1.  crontab -u wwwrun -e
  2. This brings up an editor (vim) for the cron jobs assigned to the user wwwrun (which is the account nextcloud runs under). The editor commands are somewhat cryptic...to go into insert mode, press i and the cursor will move to the top of the screen
  3. Enter (or cut and paste) the following: */5  *  *  *  * php -f /srv/www/htdocs/nextcloud/cron.php
  4. Do not press Enter! 
  5. Now you are done, press Escape to get out of editing mode and then :q to quit and save.
  6. This will have scheduled the Nextcloud housekeeping to run every 5 minutes
Now, we need to tell Nextcloud that what has been done.
  1. In a browser, log into Nextcloud as the admin and navigate to the Settings | Basic Settings screen
  2. In the top Background Jobs section, click on Cron (you have to enter the admin password again to confirm).
Now it makes sense to update the Nextcloud config to ready for external access.
  1. Navigate to /srv/www/htdocs/nextcloud/config/ and open config.php in kate (or your text editor of choice).
  2. In the trusted domains_array add some entries after 0 => 'localhost', so that the Nextcloud server will work when accessed via other URL's besides localhost. I would suggest the following (adapted to your setup)...
    1. 1 => 'the-ip-address-of-the-server',
    2. 2 => 'servername.domain.com',
    3. 3 => 'servername.localdomain',

Saturday 26 January 2019

NextCloud on OpenSuse 15.0

Most of the install guides (https://en.opensuse.org/SDB:Nextcloud and https://en.opensuse.org/SDB:LAMP_setup) use the command-line but a lot can be done using Yast. Do this logged on as root or else you will need to enter the root password a lot and preface command line instructions with sudo.
  1. Get the LAMP stack is installed and running...
    1. Yast | Software | Software Management | View | Patterns
    2. Tick Web and LAMP Server under Server Functions and press Accept
    3. Yast | System | Services Manager
    4. Select apache2  set the Start Mode to On Boot, and then press Start to start it immediately. Repeat for mariadb. Then press Ok.
  2. Get MariaDB set up properly...
    1. Fire up a terminal session and run mysql_secure_installation
    2. Press enter since there is no current MariaDB root password and then select Y to set one - enter your new MariaDB root password twice
    3. Select Y for the rest of the options
  3. Set up a database for Nextcloud...
    1. Fire up a terminal session (or keep using the previous one) and run mysql -u root -p
    2. You will be prompted for the password you entered above
    3. To create the database type create database nextcloud; and then press Enter. Don't forget the semicolon at the end.
    4. Then create a user ID and password for Nextcloud to access the database. Type create user ncdbuser@localhost identified by 'your-password-here'; and then press Enter
    5. Now give the Nextcloud ID access to the database. Type grant all privileges on nextcloud.* to ncdbuser@localhost identified by 'the-same-password-above'; and then press Enter
    6. Now type exit; and press Enter to finish
  4. Enable PHP so that Nextcloud can run
    1.  Fire up a terminal session (or keep using the previous one) and run a2enmod php7
    2. Then restart the apache webserver with apachectl restart
  5. Now install Nextcloud. The version that install with OpenSuse 15.0 is 13.0.2 but, as of writing 13.0.8 is in the update repository. If you have a vanilla install of OpenSuse the Update Repositories should already added to the Software Management module.
    1. Yast | Software | Software Management | Search | nextcloud
    2. Tick nextcloud and then press Accept
    3. A whole load of dependencies will display, press Continue
  6. Now configure Nextcloud.
    1. Fire up a web browser and navigate to locahost/nextcloud
    2. You should see a login screen as asking you to create an admin account and password for Nextcloud. Enter a new user ID and password here (these should be different from the previous userID's and passwords). DO NOT PRESS the Finish Setup button yet!
    3. Select MySQL as the database and boxes for entering the details entered in section 3 will appear ("ncdbuser", "your-password", "nextcloud" and "localhost").
    4. Then you can press the Finish Setup button 
  7. You will now be logged in to Nextcloud as the admin user. Logout (option under the top right icon)
This is an older version of Nextcloud so you need to update but, at least on my install the permissions and a couple of other things are a bit messed up and the online updater won't work. To fix this, do the following in a terminal as root:
  1. cd /srv/www/htdocs
  2. chown wwwrun.www nextcloud
  3. cd nextcloud
  4. chown wwwrun.www * -R
  5. chown wwwrun.www .*
  6. chmod +x occ
  7. rmdir search
Now, in a browser, log in to Nextcloud again as the admin. Do the following...
  1. Click on the top right icon and select Settings
  2. Part way down the screen under the Version heading you should see there is a new version 14.X available (otherwise log on and off again and see if it does a rescan for updates). 
  3. Click to use the Web Updater
  4. You should then allow it to check that the updater will work (if you have reset the permissions above that should go through OK).
  5. Click to carry on with the Web Updater,  you should be taken to an update screen with a list of all the bits to be updated. This is a fresh install so they should all be fine.
  6. Press the button at the bottom to go ahead with the update 
Once that is done you will once more be logged in to the updated Nextcloud instance as an admin. Repeat the above procedure again since there will now be a 15.X update available - at the time of writing that's it, 16.X isn't out yet.

We are not done yet, although we now have Nextcloud 15.X running it now needs a bit of tuning...
  1. In the browser, go to the menu top right and select Settings. Under the Overview | Security and Setup Warnings there is a short list of things to fix...
  2. First of all get the database straightened out after the upgrades. In a terminal, logged on as root, do the following
    1. cd /srv/www/htdocs/nextcloud
    2. sudo -u wwwrun ./occ db:add-missing-indices
    3. sudo -u wwwrun ./occ db:convert-filecache-bigint
    4. The last command will warn you about taking hours, it won't since the install is fresh, so press Y
  3. When running these commands you will notice them complaining about the PHP memory limit so lets change that.
    1. In Dolphin, navigate to /etc/php7/apache2 and click on php.ini. It should open in the kate text editor. 
    2. Scroll down until you find the line memory_limit = 128M and change it to memory_limit = 512M
    3. Save the file and exit
    4. Now navigate to /etc/php7/cli and repeat the process there.
  4. Now lets set up some caching to improve performance...
    1. In Yast go to Yast | Software | Software Management | Search | apcu and check php7-APCu
    2. Then search for php7-opcache and check it
    3. Press Accept to install the packages
    4. In Dolphin, navigate to /etc/php7/apache2 and click on php.ini. It should open in the kate text editor. 
    5. Scroll down until you find the [opcache] section. Remove the leading semicolon and edit (if required) the following lines:
      1. opcache.enable=1
      2. opcache.enable_cli=1
      3. opcache.memory_consumption=128
      4. opcache.interned_strings_buffer=8
      5. opcache.max_accelerated_files=10000
      6. opcache.revalidate_freq=1
      7. opcache.save_comments=1
    6. Now add a new section [apc] at the end with the line apc.enable_cli=1
    7. Save the file and exit
    8. Now navigate to /etc/php7/cli and repeat the process there.
    9. Now navigate to /srv/www/htdocs/nextcloud/config/config.php and open it in the kate editor
    10. Just before the last line add in  'memcache.local' => '\OC\Memcache\APCu',
    11. Save the file
    12. Restart the web server in a terminal using apache2ctl restart
  5. Log on to Nextcloud and there should be a lot less things in the Settings | Overview. I don't care about the CalDav and CardDav issues since it only really affects Apple stuff - which I don't have any of.
However, SSL really does need to be set up, but that is for another posting...

One last thing. I just had a look in the logs and it was full of error messages about using "...a fallback version of the intl extension". Fire up Yast, install php7-intl and then restart apache (again). Problem solved.