cryptsetup close – not working for LVM on LUKS – device busy

I am working right now on some blog posts on a full dmcrypt/LUKS-encryption of Laptop SSDs. Whilst experimenting I stumbled across the following problem:

Normally, you would open an encrypted device by

cryptsetup open /dev/YourDevice cr-YourMapperLabel

(You have to replace the device-names and the mapper-labels by your expressions, of course). You would close the device by

cryptsetup close cr-YourMapperLabel

In my case I had created a test device "/dev/sdg3"; the mapper label was "cr-sg3". However, whenever I tried to close this special crypto-device I got the following message:

mytux:~ # cryptsetup close  cr-g3
Device cr-g3 is still in use.

Ops, had I forgotten a mount or was something operating on the filesystem inside cr-sg3 still? Unfortunately, lsof did not show anything. No process seemed to block the device ...

mytux:~ # lsof /dev/mapper/cr-g3
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1111/gvfs
      Output information may be incomplete.


mytux:~ # dmsetup info -C  
Name                Maj Min Stat Open Targ Event  UUID                                                                
cr-g3               254  16 L--w    2    1      0 CRYPT-LUKS1-Some_UUID_Info-cr-g3                  

Note the "2" in the Open column. Clearly, there was a count of 2 of "something" that had a firm grip on the device. I also tried

mytux:~ # dmsetup remove -f cr-g3
device-mapper: remove ioctl on cr-g3  failed: Device or resource busy
Command failed

After this trial the dmsetup table even showed an error:

mytux:~ # dmsetup table
cr-g3: 0 218034943 error 

What the hack? On the Internet I found numerous messages related to similar experiences - but no real solution.

Whatever problems these guys had - I tested with other quickly created encrypted filesystems. In most cases I had no problems to close them again. It took me a while to figure out that the cause of my problems was LVM2! Stupid me! Actually, I had defined a LVM volume group "vg1" inside the encrypted partition - on purpose. The related volumes could, of course, be seen in the "/dev/mapper/" directory

mytux:~ # la /dev/mapper 
total 0
drwxr-xr-x  2 root root     440 Nov  8 15:53 .
drwxr-xr-x 28 root root   12360 Nov  8 15:53 ..
crw-------  1 root root 10, 236 Nov  8 15:46 control
lrwxrwxrwx  1 root root       8 Nov  8 16:12 cr-g3 -> ../dm-16
lrwxrwxrwx  1 root root       8 Nov  8 15:54 vg1-lvroot -> ../dm-17
lrwxrwxrwx  1 root root       8 Nov  8 15:53 vg1-lvswap -> ../dm-18

and also in "dmsetup"

mytux:~ # dmsetup info -C  
Name                Maj Min Stat Open Targ Event  UUID                                                                
vg1-lvswap          254  18 L--w    0    1      0 LVM-Some_UUID_Info
vg1-lvroot          254  17 L--w    0    1      0 LVM-Some_UUID_Info
cr-g3               254  16 L--w    2    1      0 CRYPT-LUKS1-Some_UUID_Info-cr-g3                  

So do not let yourself get confused by the fact that "lsof" may not help you to detect processes which access the crypto devices in question! It may be the kernel (with its LVM component) itself! (triggered by udev ...). This problem will always appear when you work with LVM on top of LUKS. It also explained the count of 2. The solution was simple afterwards:

mytux:~ # vgchange -a n vg1
  0 logical volume(s) in volume group "vg1" now active
mytux:~ # cryptsetup close cr-g3
mytux:~ # 

Always remember to explicitly deactivate LVM volume groups that may exist on LUKS encrypted partitions before closing the crypto-device! Have a look with the commands "lvs" or "vgdisplay" first!

Addendum 12.11.2018:
The last days I have read a bit more on crypto-partitions and volumes on the Internet. I came across an article which describes the deactivation of the volume groups before closing the crypto-device clearly and explicitly: