File control: fcntl raw and higher-level support. #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

milseman wants to merge 27 commits into apple:main from milseman:record_authority

Contributor

milseman commented Apr 6, 2021

Add in support for fcntl and all the raw commands and flags it supports. Add in Swiftier representations of common and important invocations.

System Sketch

New design patterns:

We provide rawer, but complete, FileDescriptor.fcntl() and FileDescriptor.Command interfaces and wrappers.
We provide Swiftier type-safe interfaces for a common and important subset.
FileDescriptor.FileLock is our first RR wrapper of a real C struct.

Externals:

PID, and PIDOrGPID.
flock to coordinate design with get/setLock().
fsync to coordinate design with F_FULLFSYNC etc.

Unknowns:

Headers have even more locking commands/variants, should we promote these to the Swifty APIs?
Is Errno.resourceTemporarilyUnavailable too removed from EAGAIN? Should we introduce Errno.again alias?
Are there other Swiftier interfaces we should provide?

Testing needs:

Normal mocking tests
Sample code, especially for provided Swiftier interfaces

milseman requested a review from lorentey

April 6, 2021 15:32

milseman marked this pull request as draft

April 6, 2021 15:35

Contributor Author

milseman commented Apr 6, 2021

In light of the.... poorly imagined aspects of POSIX fcntl locks, we might drop the Swiftier lock interface, pending on how flock turns out.

lorentey previously approved these changes

View reviewed changes

Sources/System/FileControl.swift Outdated Show resolved Hide resolved

Sources/System/FileControl.swift Outdated Show resolved Hide resolved

Sources/System/FileControl.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated

+                  ///
+                  /// The corresponding C constant is `F_FULLFSYNC`.
+                  @_alwaysEmitIntoClient
+                  public static var fullFsync: Command { Command(F_FULLFSYNC) }

Member

lorentey May 10, 2021

What are we going to call fsync(2)? I think I'd suggest FileDescriptor.sync or synchronize. We should use the same name here.

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

weissi reviewed

View reviewed changes

Sources/System/FileControl.swift Outdated Show resolved Hide resolved

weissi reviewed

View reviewed changes

Sources/System/FileControlRaw.swift Outdated

+                  @_alwaysEmitIntoClient
+                  public static var append: StatusFlags { StatusFlags(O_APPEND) }
+                  /// Enable the SIGIO signal to be sent to the process

Member

weissi May 16, 2021

does anybody still use this? Do we really want to offer it?

weissi reviewed

View reviewed changes

Sources/System/FileControlRaw.swift Outdated Show resolved Hide resolved

milseman mentioned this pull request

API Roadmap #16

Open

milseman dismissed lorentey’s stale review via

4c5e883

September 1, 2021 15:26

milseman added 20 commits

October 12, 2022 10:13


          Add advisory file locking API

eb5dd59


          wip: fcntl

ec315e9

wip

d669aba


          wip: ofd locks

65bd8f8


          wip: linux ofd

6e96a7a


          wip: builds

f66e6ec

wip

0baa67d


          wip: ofd locks working

1fa72c3


          wip: cleanup

df8f085


          wip: removing fcntl API

64040f3


          wip: drop more fcntl API from this PR

b32fa0e


          wip: more cleanup

300188a


          wip: drop overloads, fix docs

120f8b3


          wip: sendable

d05c4b9


          lock/tryLock/unlock

c2ccb54


          lock and lock(wait:)

a1cbeeb


          Merge remote-tracking branch 'origin/main'

a15d9a7


          workaround SIL verifier

015ba24


          fix linux build

9b3d73f


          Update test manifest for linux

e8eb2ed

milseman force-pushed the record_authority branch from 4c5e883 to e8eb2ed Compare

March 8, 2023 16:49

milseman marked this pull request as ready for review

March 8, 2023 16:49

Contributor Author

milseman commented Mar 8, 2023

Rebased on top of #111 and closer to being merge-worthy.

Needs some simple tests, some differences between Linux and Darwin dissected, and version annotations.

milseman changed the title ~~[SKETCH] Add in fcntl raw, and higher-level support.~~ File control: fcntl raw and higher-level support.

milseman requested a review from lorentey

March 8, 2023 16:51


          Fixup comments and names

aea7ca4

Contributor Author

milseman commented Mar 8, 2023

Also, we need to probably figure out some kind of fallocate(2) API to unify with F_PREALLOCATE's struct fstore.

milseman added 2 commits

March 8, 2023 10:21


          Linux cleanup

6bee7a2


          Fixups for Linux

Contributor Author

milseman commented Mar 8, 2023

@swift-ci please test

milseman added 4 commits

March 12, 2023 09:26


          wip: add some more raw types, reorganize code somewhat

547a1ff


          Update comments

979cb84


          Add in raw structs to finish fcntl for Darwin

d3bcf73


          comments

dc4270d

Contributor Author

milseman commented Mar 16, 2023

Early draft of API doc: https://gist.github.com/milseman/2e59c10299f04c682c4090c4d2ecc0e0

etcwilde pushed a commit to etcwilde/swift-system that referenced this pull request


          [Deque] Work around stdlib issue with Array.withContiguousStorageIfAv…

de5f1e6

…ailable (apple#44)

Resolves apple#27.

etcwilde pushed a commit to etcwilde/swift-system that referenced this pull request


          BitArray and BitSet data structures (apple#83)

173d2ac

* Create Screen Shot 2021-05-17 at 7.15.09 PM.png

* Delete Screen Shot 2021-05-17 at 7.15.09 PM.png

* Create BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Added BitArray as a product to the package

* Update Package.swift

* Update Package.swift

* Update Package.swift

* Update BitArray.swift

* Update Package.swift

* Update Collections.swift

* Update Collections.swift

* Update Package.swift

* Create BitArray.xcscheme

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* build: add a CMake based build system (apple#42)

Add a CMake based build system to enable bootstrapping the toolchain
with the ability to use Swift Collections in SPM.

* Update BitArray.swift

* Update BitArray.swift

* git: ignore vim swap files (apple#41)

* Initial conformation to Collection. Lots of faulty code right now

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* Update BitArray.swift

* [Deque] Work around stdlib issue with Array.withContiguousStorageIfAvailable (apple#44)

Resolves apple#27.

* [manifest] Ignore CMake files (apple#47)

The package manager doesn’t like it when it finds unexpected non-source files in source directories.

* [meta] Update dependencies (apple#48)

(This only affects local development.)

* [Deque][NFC] Remove unused code (apple#49)

* Changes to support OpenBSD. (apple#52)

* [cmake] Support amd64 architecture name.

On OpenBSD, "x86_64" is called "amd64", and therefore recognition of this
name is required for the cmake build.

* [cmake] Add missing file.

This is required since, for example, `Deque+Extras` requires
`_withContiguousStorageIfAvailable_SR14663`.

* ManagedBuffer.capacity is unavailable on OpenBSD.

ManagedBuffer requires nonstandard API to implement `capacity` and
subsequently is unavailable on OpenBSD. For portability, just use
`minimumCapacity` on this platform.

* Fixed subscript to have bit manipulations

* Update BitArray.swift

* Update BitArray.swift

* Moved Collection Conformance into appropriate file

* Implements some bit manip

* Fixed issue with endIndex resetting when storage element is full

When an element of the storage array in the BitArray filled, the endIndex would reset to 0, but then would be the correct value at any other time. This was due to a calculation error in endIndex

* Update invalid OrderedDictionary invariant check. (apple#53)

* Added XCTests, MutableCollection Conformance, Fixed endIndex bug

MutableCollection conformance still has bugs to be fixed
Additionally, the code is still very rough
endIndex had an issue where it was 0 when the array was empty. Fixed this by switching the code for count and endIndex, and making endIndex = 1 when count is 0

* Update BitArrayTests.swift

* Fixed subscript bug

Wouldn't work sometimes when current value was true because the return for `storage[index] & mask` could be anything >= 1, and not == 1, hence resulting in a unwanted toggle of the value. Fixed by comparing a boolean value of the current value to the boolean value of the newValue instead of the integers

* Code clean up ish

* Fixing a small typo on quick help documentation. (apple#55)

* Modifying Existing Code From Feedback

- `endIndex` corrected
- changed `remaining` in `count` to use ternary operator instead of being a computed local variable
- added `_split` as a helper function to `subscript`
- modified `set{...}` portion of `subscript` and included comments of the different options
- removed `clear()` as `removeAll()` seems to already exist somewhere
- modified unit tests to have a new BitArray for every test instead of a shared one
- Improved `testSubscriptSetOnly()` to hopefully be more thorough

* Added Bitwise Operations (No Tests)

Both mutable and immutable OR, AND, XOR operations

* Mark OrderedSetDiffingTests with availability (apple#54)

* Added Rough Bitwise Operation UnitTests

* Added BitSet Files

Did not add methods as I know a lot will go in BitStorage, which Im still figuring out

* Beginning implementations for BitSet

* Mutating Bitwise Operations

Performance-UNfriendly starter implementations

* Fixed subscript set mask for the &= operation and changed fatalErrors to preconditions

changed mask on &= operation from 0 to ~mask

* Also corrected subscript set test to fail if mask of 0 was used (instead of ~mask)

* Implemented BitSet with BitArray storage type and some tests

Tests are not yet passing. Pushing early so mentor can see progress
So sorry for waiting to push so late!

* Fixed error in first BitSet test ( which is testAppend() ) so it passes!

* BitSet formUnion initial implementation

* formUnion() test added and implementation modified to pass test

formUnion() is still incomplete and needs more tests atm

* Finished formUnion() for BitSet

* Modified formUnion() to use bit shifts instead

* Initial formIntersection function for BitSet

Untested

* Added an array view for BitSet, fixed BitSet.count bug, added and modified tests

* Initial Cartesian Product implementation and unfinished test

* Update BitArrayConformances.swift

* [CMake] swift-collections → SwiftCollections (apple#60)

* [CMake] swift-collections → SwiftCollections

This is an integration FIXME for SwiftPM: swiftlang/swift-package-manager#3595 (comment)

* [CMake] Generate exports under cmake/modules/SwiftCollectionsConfig.cmake

* Work around another MergeModules crash (apple#62)

* [utils] Add utility for shuffling sources to help reproduce MergeModules issues

* [Deque] Collapse Sequence/Collection conformances into a single file

The MergeModules phase in the compiler is broken: it isn’t always able to correctly handle circular dependencies across source files, like the ones that keep organically popping up with Sequence/Collection conformances that are defined in separate files.

Such dependencies sometimes triggers a compiler crash, depending on some (unknown) environmental parameters.

Trying to manually find and eliminate these circular dependencies has not proved productive. Instead, roll all Collection conformances into a single, huge source file.

* [Deque] Update CMakeLists

* [test] checkCollection: Don’t pass decreasing indices to distance(from:to:) (apple#63)

Also, try running the conformance checkers on the minimal collection types to catch similar issues.

* Attempting to make an Index

* First steps in implementing Index struct for BitSet

* Fixed up endIndex

* Last tweaks of tonight

Tomorrow: fix up Index implementations, SetAlgebra, Unit Testing

* Adding index function preconditions

* Altered index(_ index: Index, offsetBy distance: Int) -> Index

* Update BitSetConformances.swift

* Update BitSetBitwiseOperations.swift

* Time for SetAlgebra!

* Transitioning to using CollectionsTestSupport

* Conformance to Set Algebra (without bringing in bitarray methods)

* Overloaded Bitwise Operators on BitArray

* Tilday (bitwiseNOT operations)

* Fixed coding style - 2 space indents

* Initial implementation od initializers

Bug found

* Fixed count bug

* Completed and XCTested BitArray initializers

By"XCTested" I mean using my own XCTests without the thorough CollectionsTestSupport

* Update BitArray.swift

* BitSet Initializers -- XCTested

By "XCTested" I mean not thoroughly tested with the CollectionsTestSupport methods

* BitSet's "custom" SetAlgebra

* contains(Int) and dropExcessFalses() methods for BitSet

* Consistently use UNIT.bitWidth instead of the number 8

This will be useful for benchmarking purposes

* Modified BitArray.swift according to feedback

Also added more typealias consistencies (using `UNIT.max` instead of hardcoded 255)

* Modified BitArray bitwise operations according to feedback

Added overloaded mutating operators

* Modified BitArrayConformances.swift according to feedback

* Modified BitArrayOperations.swift according to feedbacl

* Adjust Bitwise Operations' naming to be consistent with that of the atomics package

* .

* Modified According to Feedback

* testing

* First working test using CollectionsTestSupport

Testing sequence initializer

* All initializer tests for BitArray (except using BitSet)

* Update BitArrayTests.swift

* BitArray Append test

* Improved BitArray append() test to test multiple appends

* Untested remove functions

Also updated any UInt8 usages to use the UNIT typealias instead

* BitArray removeLast() Test

* Update BitArrayTests.swift

* BitArray removeAll() test

* BitArray removeLast(_ rangeSize: Int) function test and bug fix

* BitArray removet(at: Int) tests

Also added test for discardable results for current and existing remove tests

* BitArray removeFirst(_ rangeSize: Int) test implemented and passing

Current method has terrible run time for now as I can't get the optimized version to work yet

* BitArray first and last true index functions TEST and BUG FIX

firstTrueIndex()
lastTrueIndex()

* Failing OR and AND bitwise operations test on BitArray

* Update BitArrayTests.swift

* Update BitArrayTests.swift

* Update BitArrayTests.swift

* PUSHHHHH

* removeFirst(_ rangeSize: Int) optimization/bugFix

* BitArray bitwise operations tests and bug fixes

* BitSet tests, bug fixes, and name changes

sequence init test and bug fix
change "UNIT" to "WORD"

* Update BitArrayTests.swift

* add BitSet layout to test support

* BitSet testExpressibleByArrayLiteralAndArrayLiteralInit()

* testExpressibleByArrayLiteralAndArrayLiteralInit()

* BitSet insert and contains functions test + bug fixes

* forceInsert test for BitSet

* BitSet remove function test and improvements

* Update BitSetSetAlgebra.swift

* BitSet intersection tests and bug fixes

* BitSet symmetricDifference test and bug fix

* starting index test and found interesting bug...

* BitArray init(BitSet) test

* Update BitSetTests.swift

* patch

* bug fix in BitSet index after

* BitSet index before test

* Clean up unneeded code

* Delete BitArrayBenchmarks.swift

* My added, not-working benchmark

* BitSet init from BitArray benchmark

* BiSet insert benchmark

* BitSet forceInsert benchmark

* Passing BitSet index offset by test

I plan to add more to the test to test offsets from indices in the middle and not just start and end

* BitArray testing modification to accommodate varying WORD sizes

* Minor comment clean-up

* Correcting bitset tests for varying WORD sizes

* Headers with License info

* simple set operation comparison

incomplete

* .

* adjustments

* Update BitSetOperations.swift

* Benchmarks and preconditions

* Update BitArrayOperations.swift

* Update BitSetTests.swift

* quick edits

* .

* Documentation for bitwise operations and some initializers

* Testing `==` operator to ensure bug fix

* BitSet SetAlgebra and Operations Documentation

* Update BitSetOperations.swift

Co-authored-by: Saleem Abdulrasool <[email protected]>
Co-authored-by: Karoy Lorentey <[email protected]>
Co-authored-by: 3405691582 <[email protected]>
Co-authored-by: Vihan <[email protected]>
Co-authored-by: Rodrigo Kreutz <[email protected]>
Co-authored-by: Andrew Monshizadeh <[email protected]>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet