<div dir="ltr"><div>Thanks all for your suggestions, comments and help. I'd like to tell you that I have a satisfying resolution, but so far the root cause of my NFS woes remains a mystery.</div><div><br></div><div>The thing that really gets my goat is that, based on everything I've read following threads from Paul's very helpful ArchLinux wiki link, is that NFS buffering/caching should be transparent to the same client - the acknowledged concurrency issues that arise from the caching are stated to be a concern when multiple clients are accessing the same files. <br></div><div><br></div><div>I'll attempt to follow up on some of the ideas. TL:DR; this seems phase-of-the-moonish, and I load or latency to be the culprit at this point.<br></div><div><br></div><div>Paul said:</div><div>> Do you get permission denied errors if you run the <br>
>
below loop with 10 byte files, but run it 10000 times?
</div><div><br></div><div>Yes, with seemingly random-ish distribution. Eg. in one run it was every 100-ish file, but then a bunch of gaps of 500+ without fail, then two failures one after the other. The distribution does not seem consistent between runs.<br></div><div><br></div><div>I said:</div><div>> $<span style="white-space:pre-wrap"> <span class="gmail-il">for</span> i in {1..100}; </span><div><span style="white-space:pre-wrap">> do </span></div><div><span style="white-space:pre-wrap">> head -c 10 </dev/urandom >"/nfs_share/tmp/test_file${i}";</span></div><div><span style="white-space:pre-wrap"> > done && find /nfs_share/tmp/ -name test_file* -exec mv {} /nfs_share/final_location \;</span></div><div><span style="white-space:pre-wrap"><br></span></div>
</div><div>Bob said:</div><div>>
Right, so in 'command1 && command2' if command1 returns non-zero then command2 will not run.<br>
><br>
</div><div>> OTOH, [Adam] is getting permission failures on the mv command, implying that the part before the && is returning 0.</div><div><br></div><div>You touched an important point here Bob. Without realizing it, I was making some assumptions</div><div> 1. That each command in the for loop runs sequentially</div><div> 2. That there will be no failures creating the files</div><div> 3. That before each `head` command exits, the redirected output file is fully written (as far as the filesystem user is concerned) and closed.</div><div><br></div><div>Something like this would have expressed my intentions more correctly<br></div><div> $<span style="white-space:pre-wrap"> <span class="gmail-il">for</span> i in {1..100}; </span><div><span style="white-space:pre-wrap"> do </span></div><div><span style="white-space:pre-wrap"> head -c 10 </dev/urandom >"/nfs_share/tmp/test_file${i}" && mv <span style="white-space:pre-wrap">"/nfs_share/tmp/test_file${i}</span>" <span style="white-space:pre-wrap">/nfs_share/final_location</span>;</span></div><div><span style="white-space:pre-wrap"> done</span></div>
</div><div><br></div><div>In the end, I think both give useful information. If the issue had only to do with the time between the end of a file being written and the attempt to unlink it, I'd expect the errors to be biased toward the end of the set in the `find` version. However, the errors seem to be randomly distributed throughout the set.<br></div><div><br></div><div>Bob said<br></div><div>> [Adam], do you get the failures if you create just a single file of 10,000 bytes?</div><div><br></div><div>Not usually. If I manually run `head -c 10k </dev/urandom >/nfs_share/tmp/test_file && mv /opt/odrive/tmp/test_file /<span style="white-space:pre-wrap">nfs_share/final_location</span>`, I occasionally get an error. It seems like this is more likely to happen if I run the loop, interrupt when an error pops up, then immediately do the one-off copy. Manually generating 100k files I got the error a few times, say trial 4 and trial 7, then trial 15, then couldn't reproduce after maybe 50 more tries.<br></div><div><br></div><div>I'll update if I find anything new. It may be a while, as I need to construct a useful ticket for the folks who have access to the logs, and it's not a top priority.</div><div><br></div><div>Thanks again and happy new year,<br></div><div>Adam<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Dec 17, 2022 at 7:04 PM Khalid Baheyeldin <<a href="mailto:kb@2bits.com">kb@2bits.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Adam,</div><div><br></div><div>Is NFS a must in this case, or can you use other file systems?</div><div><br></div><div>The reason I suggest this is that historically NFS have had a problematic <br></div><div>legacy in certain use cases. These include caching inconsistencies in some</div><div>use cases. <br></div><div><br></div><div>About a decade ago I investigated how to share file systems across two web <br></div><div>hosts. That was for a web site to share the code and static files on two web <br></div><div>servers. NFS was the most problematic, and not practical. <br></div><div><br></div><div>There was GFS (not GFS2) and GlusterFS which showed promise last I looked <br></div><div>at filesystems for the above use case. <br></div><div><br></div><div>For certain types of use, sshfs may be all you need (e.g. mounting server shares <br></div><div>to my laptop). Have been using it for years without issues. It even transparently</div><div>reconnects after the laptop wakes up from sleep. It does not need a daemon on</div><div>the server (openssh server does it all), which means great security. <br></div><div><br></div><div>All you need is install the sshfs package on the client, and run the following command:</div><div><br></div><div>sudo sshfs -o $OPTS user@host:/remote/file/system /local/mount</div><div><br></div><div>The OPTS variable is where all the magic is</div><div><br></div><div>IdentityFile=~myname/.ssh/id_ed25519,port=22,follow_symlinks,allow_other,noatime,</div><div>idmap=file,uidfile=/etc/sshfs-uids,gidfile=/etc/sshfs-gids,nomap=ignore,</div><div>workaround=rename,nonempty,reconnect,ServerAliveInterval=3,</div><div>ServerAliveCountMax=10</div><div><br></div><div>Besides your ssh private key, two files are needed to map the uid and gid to what they <br></div><div>are on the server:</div><div><br></div><div>user1:1001</div><div>user2:1002</div><div><br></div><div>users:100</div><div>user1:1001</div><div>user2:1002</div><div><br></div><div>And that is all there is to it.</div><div><br></div><div>SSHFS would not work in a high load scenario though. <br></div></div>
_______________________________________________<br>
kwlug-disc mailing list<br>
To unsubscribe, send an email to <a href="mailto:kwlug-disc-leave@kwlug.org" target="_blank">kwlug-disc-leave@kwlug.org</a><br>
with the subject "unsubscribe", or email<br>
<a href="mailto:kwlug-disc-owner@kwlug.org" target="_blank">kwlug-disc-owner@kwlug.org</a> to contact a human being.<br>
</blockquote></div>