PC Ekspert Forum - RAMDisk - Ubrzavanje kompa uz pomoc prebacivanja cache/temp foldera na RAM Disk

PC Ekspert Forum (https://forum.pcekspert.com/index.php)

- Aplikacije (https://forum.pcekspert.com/forumdisplay.php?f=37)

- - RAMDisk - Ubrzavanje kompa uz pomoc prebacivanja cache/temp foldera na RAM Disk (https://forum.pcekspert.com/showthread.php?t=322514)

Citiraj:

Autor calypso (Post 3801169)

Gledam CPU usage dok testiram sa ATTO Disk Benchmarkom, nema nikakvih spikeova niti mi zakucava jednu jezgru, malo podigne frekvenciju procesora (sa 1GHz na 1.5GHz u prosjeku) i to je to... Znaci doslovno nista se ne desava...

Load je i tako vjerojatno raspoređen između jezgri. Dodatno, pretpostavljam da moderni procesori nemaju previše problema s rješavanjem takvih zadataka, čemu jamačno u prilog ide relativno veliki cache.

Što se tiče LLMa, isti i tako ide u VRAM, što bi ga trpao u RAM disk?

Citiraj:

Autor mkey (Post 3801220)

Ne znam, zato pitam... U slucaju da brlja po disku sa nekim temp glupostima, to bi moglo ici u RAM... Uostalom, prenio sam sto mi je DeepSeek rekao, ako on laze, lazem i ja... :rtfm: :D

A i ovo za VRAM, mislim da to funkcionira jedino ako imas nVidiju... Integrirke ionako koriste DRAM... :) A uvijek se nadje netko tko zeli korisiti lokalni AI na svojoj masini, pa makar za probu, pa eto, nema nVidiju, ima samo integrirku i... Ma nek se igra i veseli... :)

Koliko ja znam, ovo što sam čačkao po "AI" lokalno (InvokeAI uglavnom) bitan je samo VRAM, proizvođač GPU je nevažan što se toga tiče. Ako model može biti skroz učitan u VRAM dobiješ jednu brzinu, ako ne može biti učitan dobiješ drugu brzinu i vjerojatno rušenje.

InvokeAI ima neki workaround koji bi kao trebao spriječiti rušenje pa koristi RAM ali to uz velike penale.

Ako uz to imaš nevidiju onda dobiješ i compute boost od kakvih 300-400% prema AMD, makar prema seriji 7xxx koju sam ja ponešto testirao. 3060 Ti 8GB je bila od 7800 XT 16GB podosta brža u computu, a te dvije nemaju veze jedna s drugom, ako gledaš raster. Sve dok koristiš model koji stane u 8GB, jelte.

Priupitao sam statisticko pogladjalo #2 (aka. DeepSeek) zasto su performanse FAT32 volume vece nego NTFS u RAMDisku i... pa mogu reci da mi je dao OK rezultate, na tragu onoga sto sam i sam zakljucio...

Kada su stvari toliko brze kao u RAMu, onda svaki overhead igra ulogu i vidljiv je...

Your observation that FAT32 performs faster than NTFS on a RAMDisk in ATTO Disk Benchmark (or similar tools) can be explained by several technical factors. Here’s why:

1. Overhead of NTFS Features
- NTFS includes advanced features that add overhead, which isn’t beneficial (and can even slow things down) in a RAMDisk environment:
- Journaling (for crash recovery) → Writes extra metadata.
- File permissions (ACLs) → Unnecessary for a RAMDisk.
- MFT (Master File Table) updates → More complex than FAT32’s simple allocation table.
- Small file handling → While NTFS is better for small files on HDDs/SSDs, RAM is so fast that the overhead of NTFS’s metadata management can actually slow it down.

FAT32, being much simpler, has:
- No journaling.
- No file permissions.
- A straightforward File Allocation Table (FAT) structure.
→ Less CPU overhead = faster in RAM.

2. Benchmark Workload Matters
ATTO Disk Benchmark typically tests sequential read/write speeds with different block sizes. In this case:

- For large sequential transfers, both NTFS and FAT32 should be close to RAM’s raw speed (~10+ GB/s).
- For small block sizes (4KB–64KB), FAT32 often wins because:
- NTFS has to update more metadata (MFT, journals, etc.).
- FAT32’s simple structure allows near-direct writes.

3. RAMDisk Optimizations
- NTFS is optimized for disks (HDD/SSD), not RAM.
- FAT32’s simplicity aligns better with RAM’s speed because:
- No need for fragmentation management (RAM is inherently fast at random access).
- No need for advanced caching (RAM is already cache).

4. File System Drivers & Caching
- The NTFS driver in Windows is more complex and may introduce tiny delays.
- The FAT32 driver is lightweight and well-optimized for simple operations.
- Windows caching mechanisms may behave differently between the two.

When Would NTFS Be Faster on RAMDisk?
- If you’re working with many tiny files, NTFS’s MFT might help (but often, FAT32 still wins).
- If the workload involves heavy metadata operations (though RAM is so fast that the difference is minimal).

Pa eto, rezultati ne lazu, svi testovi koje sam radio su radjeni na isti nacin, sa istim volume sizeom, samo sam mijenjao filesystem...

Jedino, ne kuzim ovo za FAT32 vs NTFS performanse jer mi i DeepSeek veli da je NTFS brzi... Valjda desetljecima zivim u uvjerenju da je FAT32 brzi bas zbog minimalnog overheada i jednostavnosti samog FSa... Tako da, sljedeci projekt ce biti da istestiram FAT32 vs NTFS na nekom NVME disku i SSDu pa da vidim sta je sa time... Budem si samo pronasao neki laptop na kojem cu to moci bez problema odraditi...

Baš se igramo s ollamom na poslu s nekim modelima. Čim ponestane VRAMa i krene koristiti sistemski RAM stvar se strašno uspori.

OK, konačno došlo moje na red. Kao izvor za kopiranje sam napravio fajlove random (binarnog) sadržaja i smjestio ih na NVME A:

1. 100k fajlova od 1 KiB
2. 10k fajlova od 4 KiB
3. 1k fajlova od 10 KiB
4. 1k fajlova od 1 MiB

Test sam napravio u 4 koraka, ponovio par puta s ponovljivim rezultatima:

1. rsync direkt u RAM preko TMPFS
2. rsync na BTRFS RAM particiju, bez kompresije
3. rsync na BTRFS RAM particiju, sa kompresijom
4. rsync na BTRFS particiju s kompresijom na NVME B

Ukupno za kopiranje 112k fajlova ukupne veličine 1.19 GiB (na kraju vidim da je tu bilo i nešto više fajlova, ali mislim da nije bitno s obzirom da samo to i tako napravio nasumično).

1. kopiranje direkt u RAM: 189 MiB/s
2. kopiranje na RAM BTRFS particiju bez kompresije: 74 MiB/s
3. kopiranje na RAM BTRFS particiju s kompresijom: 79 MiB/s
4. kopiranje na BTRFS particiju na drugom NVME: 98 MiB/s

Pod 3 pretpostavljam kako je limitirana mogućnost komprimiranja s obzirom na nasumičan sadržaj fajlova. Da su neki tekstualni fajlovi vjerojatno bi ta brzina bila veća.

Mislim da se ovdje pokazalo ono što je bilo i za očekivati, kod rada s nekim povećim arhivama RAM kao temp dir već i na DDR4 ima smisla.

Code:

# create a bunch of small files; 100k 1k large files

for i in {1..99999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    # echo "1K#$pad$i";

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/1K#$pad$i bs=1K count=1

done;



# create a bunch of small files; 10k 4k large files

size="4K"

for i in {1..9999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# create a bunch of small files; 1k 10k large files

size="10K"

for i in {1..999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# create a bunch of small files; 1k 1m large files

size="1000K"

for i in {1..99}

do

# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

rm -f /dev/shm/btrfs.fs

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# copy directly to RAM

mkdir /dev/shm/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /dev/shm/RAMDisk # 189 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.782 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052

# 

# sent 1,285,676,660 bytes  received 2,149,052 bytes  198,127,032.62 bytes/sec ~ 189 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /de

# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

rm -f /dev/shm/btrfs.fsv/shm/RAMDisk



# create a btrfs filesystem, without compression

mkdir /mnt/btrfs

dd if=/dev/zero of=/dev/shm/btrfs.fs bs=1M count=2024

mkfs.btrfs /dev/shm/btrfs.fs

mount /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 74 MiB/s ; heavy CPU usage: gvfsd-trash



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.805 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  78,050,043.15 bytes/sec ~ 74 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

rm -f /dev/shm/btrfs.fs

Citiraj:

Autor mkey (Post 3801606)

Code:

# create a bunch of small files; 100k 1k large files

for i in {1..99999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    # echo "1K#$pad$i";

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/1K#$pad$i bs=1K count=1

done;



# create a bunch of small files; 10k 4k large files

size="4K"

for i in {1..9999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# create a bunch of small files; 1k 10k large files

size="10K"

for i in {1..999}

do

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# create a bunch of small files; 1k 1m large files

size="1000K"

for i in {1..99}

do

# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

rm -f /dev/shm/btrfs.fs

    if [[ $i -lt  10 ]] then pad="0000"

    elif [[ $i -lt  100 ]] then pad="000"

    elif [[ $i -lt  1000 ]] then pad="00"

    elif [[ $i -lt  10000 ]] then pad="0"

    elif [[ $i -lt  100000 ]] then pad=""

    fi;

    dd if=/dev/random of=/media/mkey/data2/RAMDisk/$size#$pad$i bs=$size count=1

done;



# copy directly to RAM

mkdir /dev/shm/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /dev/shm/RAMDisk # 189 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.782 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052

# 

# sent 1,285,676,660 bytes  received 2,149,052 bytes  198,127,032.62 bytes/sec ~ 189 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /de

# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

rm -f /dev/shm/btrfs.fsv/shm/RAMDisk



# create a btrfs filesystem, without compression

mkdir /mnt/btrfs

dd if=/dev/zero of=/dev/shm/btrfs.fs bs=1M count=2024

mkfs.btrfs /dev/shm/btrfs.fs

mount /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 74 MiB/s ; heavy CPU usage: gvfsd-trash



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.805 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  78,050,043.15 bytes/sec ~ 74 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# same but with compression

umount /dev/shm/btrfs.fs

mkfs.btrfs -f /dev/shm/btrfs.fs

mount -o compress=lzo /dev/shm/btrfs.fs /mnt/btrfs



rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /mnt/btrfs # 79 MiB/s



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.773 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  83,085,529.81 bytes/sec ~ 79 MiB/s

# total size is 1,278,858,240  speedup is 0.99





# copy from one NVME to another NVME; with BTRFS compression

mkdir /home/mkey/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /home/mkey/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 0.788 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  103,026,056.96 bytes/sec ~ 98 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /home/mkey/RAMDisk

umount /dev/shm/btrfs.fs

 rm -f /dev/shm/btrfs.fs

E o ovome govorim cijelo vrijeme - real world test. Hvala mkey :) Nego jel bi ti bio bed jos sa EXT4 testirat, preferabilno sa noatime mount opcijom?

Na RAM driveu bi trebalo isključiti cache u file sistemu :D

Citiraj:

Autor mkey (Post 3801606)

Mislim da se ovdje pokazalo ono što je bilo i za očekivati, kod rada s nekim povećim arhivama RAM kao temp dir već i na DDR4 ima smisla.

Odlican test! :) Jesmo sad uspjeli dokazati da je TMP/Cache/Scratch skroz OK drzati na RAMDisku?

Pretpostavljam da ti je DDR4 u dual-channel modu, jelda?

Volio bih probati taj test na nekoj DDR5 masini, no trenutno nemam nista takvog osim mog radnog stroja koji mi treba za rad, a njega ne mislim rasturati da instaliram nativno Linux... Mozda u neku virtualnu masinu, ali to onda nije bas nativno...

Tomek, na tom drugom NVME sam napravio jednu malu EXT4 particiju i napravio rsync na nju i to je bilo ponešto sporije nego copy na BTRFS particiju, cca 74 MiB/s. To je usporedivo s brzinom koju dobivam na BTRFS particiji u RAMu. Ne da je usporedivo, nego identično :D

Skroz identičan broj daje rsync (78,050,043.15 bytes/sec) u ta dva slučaja. Sada nisam baš siguran što ja tu mjerim. Ispada da RAM BRTFS particija i fizička EXT4 particija imaju identično usko grlo, šta li?

Code:

# copy to EXT4 partition on another NVME

mkdir /media/mkey/EXT4/RAMDisk

rsync -aAHX --progress --stats /media/mkey/data2/RAMDisk/ /media/mkey/EXT4/RAMDisk



# Number of files: 112,996 (reg: 112,995, dir: 1)

# Number of created files: 112,995 (reg: 112,995)

# Number of deleted files: 0

# Number of regular files transferred: 112,995

# Total file size: 1,278,858,240 bytes

# Total transferred file size: 1,278,858,240 bytes

# Literal data: 1,278,858,240 bytes

# Matched data: 0 bytes

# File list size: 1,638,378

# File list generation time: 1.590 seconds

# File list transfer time: 0.000 seconds

# Total bytes sent: 1,285,676,660

# Total bytes received: 2,149,052



# sent 1,285,676,660 bytes  received 2,149,052 bytes  78,050,043.15 bytes/sec ~ 74 MiB/s

# total size is 1,278,858,240  speedup is 0.99



# cleanup

rm -fR /media/mkey/EXT4/RAMDisk

medo, nisam baš shvatio na što misliš. To je za slučaj kada napravim RAM BTRFS particiju?

calypso, nisam baš siguran što smo ovdje dokazali, s obzirom na ovaj gornji slučaj. Ne vidim kako u realnom scenariju podatak koji daje rsync može biti identičan na dvije decimale u dva skroz različita scenarija. U mene je DDR4 u dual channelu, jeste. Ali pitanje je što bih s TMPFS dobio na DDR5, kada je i ovako max brzina debelo ispod teoretske max brzine od samog DDR4 RAMa kada radi na 3200. Teoretski bi to bilo 25.6 GiB/s, onaj benchmark mi je pokazivao kakvih 17, a ja sa sekvencijalnim copy na TMPFS dobijem 2.5.

Citiraj:

Autor mkey (Post 3801646)

Dakle EXT4 je i dalje car Linux FS-ova po pitanju brzine. Hovewer evo jedan kontra produktivni test, jednostavan ko pasulj što bi susjedi rekli:

https://i.postimg.cc/Tw4Nb8DP/2025-0...0-Postavke.png

Testirano na Lenovo lapu u sigu, kopiranje vise fajlova razlicitih velicina sa Microna na Sandisk. Bez Ramdiska. Windows 11 sa NTFS-om, bez ikakvih optimizacija.

Citiraj:

Autor medo (Post 3801629)

Na RAM driveu bi trebalo isključiti cache u file sistemu :D

Bas mi to palo na pamet... :D Bas cu pogledati da li se moze takvo sto u Windowsima sloziti...

Citiraj:

Autor mkey (Post 3801646)

calypso, nisam baš siguran što smo ovdje dokazali, s obzirom na ovaj gornji slučaj. Ne vidim kako u realnom scenariju podatak koji daje rsync može biti identičan na dvije decimale u dva skroz različita scenarija. U mene je DDR4 u dual channelu, jeste. Ali pitanje je što bih s TMPFS dobio na DDR5, kada je i ovako max brzina debelo ispod teoretske max brzine od samog DDR4 RAMa kada radi na 3200. Teoretski bi to bilo 25.6 GiB/s, onaj benchmark mi je pokazivao kakvih 17, a ja sa sekvencijalnim copy na TMPFS dobijem 2.5.

I meni izgleda kao neko usko grlo negdje, samo gdje?

Tomek ne znam kako si došao do zaključka da je EXT4 kralj brzine, s obzirom da je u ovom mom primjeru sporiji od BTRFS 20ak%. Ili je to možda bi neki lagani sarkazam?

Također, nisam siguran koliko je primjer s tvog SSa relevantan. Ti dakle imaš 150GB u 22.051 file, odnosno skoro 7 MB po fajlu, dok je u mom primjeru 1.2GB u skoro 113k fajlova, odnosno prosječno 0.01 MB po fajlu.

Da i ne spominjem kako ne vjerujem tom njegovom mjerenju brzine a bogami niti podatku o preostalom vremenu. Čisto onako iskustveno :D

Ne budi lijen, potražio sam alternativu rsyncu u smislu kopiranja fajlova uz prikaz brzine transfera i našao ovo zgodno rješenje "tar c source | pv | tar -C x dest"

Disclaimer: koliko je ovo usporedivo s onime što rsync radi, u smislu prenošenja svih atributa, pojma nemam. Tako da ne tvrdim kako je ovo brže ili bolje od rsync, samo sam htio vidjeti što po pitanju brzine kopiranja fajlova kaže neka druga aplikacija.

Dobio sam slijedeće (za uvijek isti set fajlova):
- TMPFS: 570 MiB/s
- fizički BTRFS s kompresijom: 199 MiB/s
- fizički EXT4: 164 MiB/s
- RAM BTRFS particija, bez kompresije: 172 MiB/s
- RAM BTRFS particija, sa kompresijom: 168 MiB/s

Dakle, u odnosu na rsync sam prijenos je brži cca x2 dok je sada još i veća razlika između TMPFS i NVME. Nisam pratio, ali ovdje bih očekivao veće korištenje RAM i CPU.

Code:

# tar + pv



# to TMPFS

cd /media/mkey/data2 & tar c RAMDisk | pv | tar x -C /dev/shm/

# 1.24GiB 0:00:02 [ 570MiB/s]



# to BTRFS with compression

cd /media/mkey/data2 & tar c RAMDisk | pv | tar x -C /home/mkey/

# 1.24GiB 0:00:06 [ 199MiB/s]



# to EXT4

cd /media/mkey/data2 & tar c RAMDisk | pv | tar x -C /media/mkey/EXT4/

# 1.24GiB 0:00:07 [ 164MiB/s]



# to RAM BTRFS, without compression

cd /media/mkey/data2 & tar c RAMDisk | pv | tar x -C /mnt/btrfs

# 1.24GiB 0:00:07 [ 172MiB/s]



# to RAM BTRFS, with compression

cd /media/mkey/data2 & tar c RAMDisk | pv | tar x -C /mnt/btrfs

# 1.24GiB 0:00:07 [ 168MiB/s]



# cleanup

rm -fR /dev/shm/RAMDisk

rm -fR /home/mkey/RAMDisk

rm -fR /media/mkey/EXT4/RAMDisk

Sad mi nista nije jasno...

Ispada da je kopiranje sa jednog na drugi NVME disk brze nego kopiranje sa NVME u BTRFS RAMdisk? Ili sam ja nesto krivo shvatio?

Da, kopiranje u taj RAM BTRFS je konzistentno sporije od fizičkog diska, što meni ima smisla. Očito ima tu dosta overheada. Nisam baš primijetio da se takvo nešto uobičajeno koristi, mene je samo zanimalo što bi kompresija značila za performanse.

Ono što je tu bitno je po meni TMPFS vs NVME, a to je pak u ovom slučaju skoro 3x brže.

Citiraj:

Autor mkey (Post 3801698)

Meni je to i dalje totalno cudno jer ispada da je rad sa standardnim RAMDiskom (BTRFS) na Linuxu sporiji nego rad sa NVME diskom... Na Windowsima je to drasticno brze i na FAT32 i na NTFS... Ozbiljno, nista mi nije jasno...

Ajd pliz, ako imas vremena i volje, probaj isto to zavrtiti na EXT2 particiji - nema overheada nikakvog, najjednostavniji FS, nema journaling koji za potrebe RAMDiska i nije potreban...

Tu je poanta priče tmpfs, koji, koliko ja shvaćam, nije napravljen s performansama kao glavnim ciljem. Meni je logično da tmpfs sam mora biti brži od tmpfs + nešto. Što god da dodaš biti će veći overhead. Izuzetak bi bila možda neka situacija gdje se koristi kompresija na fajlovima koji se daju dobro komprimirati.

EXT2 RAM particija povuče s tar metodom 207 MiB/s, tako da je to 20% brže u odnosu na BTRFS.

Citiraj:

Autor mkey (Post 3801706)

Kompresija unosi IO overhead, tako da nema nikakve sanse da ce raditi brze... Mozda na manjim fajlovima i to u readu, ali kod writea je katastrofa... To sam pokazao gore na testovima na NTFSu...

Ext2 -> jos jedna potvrda da non-journaling FS ima nesto bolje performanse nego journaling, sto je ista prica i kod FAT32 vs NTFS, a sto je i logicno na kraju krajeva... Pricam o RAMDisku, nisam testirao na NVME tako da ne znam kako se to ponasa, iako i tamo ocekujem da bi FAT32 trebao biti nesto malo brzi od NTFS...

Iako, i dalje mi nije jasno zasto bi BTRFS u RAMDisku bio sporiji od BTRFS na NVME disku, to nema nikakvog smisla... Ako sam dobro shvatio, ovi tvoji testovi se svode na testiranje write performansi, zar ne? Ali svejedno, rezultati su cudni...

To je sve write, da. U testovima od prije par dana, kada sam pisao prazan od 1 GiB kojeg se dalo jeftino komprimirati, BTRFS s kompresijom je bio cca 30% brži od BTRFS bez kompresije. Penal je overhead komprimiranja, ali ipak se na kraju umjesto 1 GiB piše 70ak MiB i to sve skupa bude brže. Iako to nije primjer nečega što ima realnu primjenu, s obzirom da se radi o praznom fajlu.

Citiraj:

Autor mkey (Post 3801709)

Aha... Ima smisla, da... Trazenje optimalne velicine bloka koji bi imao najbolje performanse i kod velikih i kod malih fajlova... MS je odlucio da je to 4kB, a i meni se cini da je to OK, iako se to sve da fine-tuneat ukoliko znas koliki ce ti biti fajlovi veliki pa mozes odabrati malo veci block size da dobijes malo na performansama...

RAMDisk - Ubrzavanje kompa uz pomoc prebacivanja cache/temp foldera na RAM Disk

Citiraj:

Autor tomek@vz (Post 3801648)

Dakle EXT4 je i dalje car Linux FS-ova po pitanju brzine.

Kladio bi se da će XFS biti brži na RAM driveu pogotovo u multithreaded benchu.

Citiraj:

Autor calypso (Post 3801712)

Čitao sam negdje da je 4kB postavljeno kao optimum jer CPU komunicira s periferijom u page-ovima od 4kB.

Citiraj:

Autor medo (Post 3801733)

Kladio bi se da će XFS biti brži na RAM driveu pogotovo u multithreaded benchu.

Not a bad Idea....

@mkey - raspali :lol2:

Ne mogu reći da vidim načina kako odraditi multithreaded copy.

Citiraj:

Autor mkey (Post 3801828)

Ne mogu reći da vidim načina kako odraditi multithreaded copy.

Pokrenes 4 kopiranja istovremeno, i onda svaki proces alociras na drugi core... :D

Citiraj:

Autor mkey (Post 3801828)

Ne mogu reći da vidim načina kako odraditi multithreaded copy.

fio?

fio na tmpfs neće da može.

Iskreno, nemam pojma da li bude ovo od ikakve koristi, ali kreiral sam RAMdisk od 32GB na sveukupno 128GB DDR5-5200, prije toga pogasil sve virtualne mašine, naknadno kopiral i smjestil EndeavourOS VM od 25GB u prethodno kreirani 32GB RAMdisk i provjeril kakve su brzine pisanja i čitanja u odnosu na NVMe SSD. Swap file i ostala čuda tehnike su uključena po defaultu, a nekih konkretnih razlika na relaciji RAMdisk - NVMe SSD, u ovom mojem slučaju - baš i nema. #umount:frend:

https://i.postimg.cc/grQNgjNm/RAMdisk-01.png https://i.postimg.cc/ZWFHYcZ5/RAMdisk-02.png https://i.postimg.cc/TKXQXszs/RAMdisk-03.png

Code:

root@fedora-ws:/# df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/nvme1n1p3  237G   18G  217G   8% /

devtmpfs        4.0M     0  4.0M   0% /dev

tmpfs            62G  100K   62G   1% /dev/shm

efivarfs        128K   58K   66K  47% /sys/firmware/efi/efivars

tmpfs            25G  2.6M   25G   1% /run

tmpfs            62G  391M   62G   1% /tmp

/dev/nvme1n1p3  237G   18G  217G   8% /home

/dev/nvme1n1p2  974M  423M  484M  47% /boot

/dev/nvme1n1p1  599M   20M  580M   4% /boot/efi

tmpfs            13G  268K   13G   1% /run/user/1000

tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service

tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service

/dev/sda        3.7T  2.5T  1.3T  67% /run/media/mcg/1st DATA Backup

/dev/sdb        3.7T  2.5T  1.3T  67% /run/media/mcg/2nd DATA Backup

/dev/nvme0n1    916G  388G  482G  45% /run/media/mcg/Virtual Machines

tmpfs            13G   68K   13G   1% /run/user/0

myramdisk        32G   25G  8.0G  76% /tmp/ramdisk

root@fedora-ws:/# sudo dd if=/dev/zero of=/tmp/ramdisk/zero bs=4k count=100000

100000+0 records in

100000+0 records out

409600000 bytes (410 MB, 391 MiB) copied, 0.0846253 s, 4.8 GB/s

root@fedora-ws:/# sudo dd if=/tmp/ramdisk/zero of=/dev/null bs=4k count=100000

100000+0 records in

100000+0 records out

409600000 bytes (410 MB, 391 MiB) copied, 0.0452527 s, 9.1 GB/s

root@fedora-ws:/# sudo dd if=/dev/zero of=/tmp/zero bs=4k count=100000

100000+0 records in

100000+0 records out

409600000 bytes (410 MB, 391 MiB) copied, 0.0805062 s, 5.1 GB/s

root@fedora-ws:/# sudo dd if=/tmp/zero of=/dev/null bs=4k count=100000

100000+0 records in

100000+0 records out

409600000 bytes (410 MB, 391 MiB) copied, 0.0423088 s, 9.7 GB/s

root@fedora-ws:/#

Sekvencijalni write je i kod mene bio tu negdje, samo cca pola tvojih brojeva.

Da, sve ove novije mašine unatrag nekoliko godina se čisto dobro drže, tj. RAMdisk (više) ne dolazi toliko do izražaja.